@inproceedings{huebsch:vldb-2003,
  title={Querying the Internet with {PIER}},
  author={Huebsch, R. and Hellerstein, J.M. and Lanham, N. and
          Loo, B.T. and Shenker, S. and Stoica, I.},
  booktitle={International Conference on Very Large Databases ({VLDB})},
  pages={321--332},
  year={2003},
}

This paper presents PIER, a system for massively distributed relational database storage & querying. PIER builds on distributed hash tables to distribute & locate relational tuples across the network. The paper presents several algorithms for spreading query computations and data shipping, and the various performance results.

One of the primary motivating applications given for PIER is networking monitoring, e.g. intrusion and attack detection. Using a PIER type system for this sounds likely to generate much redundant network traffic. For examaple, querying every 15 seconds over the last 30 seconds' worth of a particular data type will involve shipping much of the same data around. This could be addressed in the monitoring application, but that's contrary to the intent of providing a basic data service, such that the application doesn't have to worry about details like that. A streaming, continuous query approach might not have the same problem.

However, the continuous approach would have its own similar issues for this application. Opposite the above problem, it might not be necessary to regularly ship data around, e.g. to investigate in detail a specific, infrequent or unique problem rather than continuous monitoring. PIER seems more appropriate for the former task. The PIER approach also seems more robust to monitor node failure, i.e. switching to a new monitor host will not require restoring history/state or waiting to collect sufficient new data.

Many of the experiments in this paper could perhaps be presented and/or conducted more rigorously. For example, many of the simulation results make no note of the number of trials conducted, and imply that given data is based on a single trial. These one-shot data points should be justified more, or multiple trials conducted & their statistical significance evaluated.

Also of note would be some discussion about the underlying networking setup. In some respects these are application details, but they do matter a great deal and could bear significant effect on observed performance. For example, is TCP used and are connections continually setup/torn down or long-lived? There are good reasons for both. If TCP is not used, how is reliability assurance & fragmentation performed? The paper mentions "dropping packets" at one point, but it's not clear what this means exactly. These choices may also have strong bearing on the accuracy of their simulation.

On a related note, the paper briefly looks into simulation over network topologies other than a complete graph. It's unclear whether cross traffic remains unmodeled in these simulations. Clearly there are topologies where congestion & collisions matter a great deal, e.g. over a (topological) bridge. It would be interesting to look at performance over specific topologies, e.g. if data and query nodes are seperated by bridge links. But, this of course tends toward a new or extended line of work investigating data placement & query host optimization to reduce latency & bandwidth.

A key detail not discussed in this paper is how all nodes storing tuples of a given relation are located and communicated with---how namespaces are used and the (high level) conduct of the multicast() operation. Following reference 18, it behaves largely as expected---namespaces (relations) are mapped onto ranges in the DHT identifier space. One interesting point here is that mapping namespaces onto high order bits could swamp that region w/ storage requests and queries on popular relations. This is a particular problem if nodes cover relatively large segments of the identifier space and node IDs aren't randomly assigned, but based on IP address or other persistent identifier. Such a scheme makes some attacks more difficult, but dooms some nodes to handling popular data & queries.

From a load balancing perspective, it makes some sense to map relations, presumably much more common than resource IDs (primary keys), onto low order bits or otherwise spread them around the network. Unfortunately, tuples then become much much harder to find. The system also most likely incurs higher latency or aggregate network bandwidth costs as a result of not grouping relations. However, effectively performing some form of load balancing does seem to be a worthwhile goal for this kind of system.