I wrote this in response to a question I got, asking to clarify my special tricks.
For a given particle and a given distance r, you are trying to count all the particles that lie between r and r+dr away. In 3D this is a spherical shell with thickness dr. To correctly normalize g(r), you have to then account for the volume contained within that shell. But, if your shell lies outside your computation box, then you have to figure out exactly how much volume lies within your box. Assuming dr is small, then really it's like computing how much surface area of the sphere of radius r lies within the box.
So, you're left with a nontrivial trigonometry problem, especially if you have a small cubical box. For most values of r, you'd have several pieces of your sphere lying outside your box, and computing the area of what's left inside would be difficult.
In 2D the math is not too hard (for a circle and a rectangular box). My trick there was to consider each quadrant of the circle separately. At most, each quadrant of the circle could intersect (or lie outside of) two edges of the rectangle. Then I could determine how much of the quadrant was within those two edges. Working it out took a few sheets of paper but the resulting formulas were fairly simple (although at this point embedded within my program and so I don't have them easily at hand).
But in 3D it's a bit harder. I cheated by forcing the program to stay away from the edges of the box in X and Y, so the only intersections I have to consider are with the faces of the box in Z. The math then wasn't too bad (to find the surface area of a spherical cap lying outside the box).
Anyway, if you have a cubical box, I don't know if there is any simple way to do it. Perhaps my trick in 2D with quadrants and two edges of the rectangle can be extended to 3D with octants and 3 faces of the cube. But you'll have to see if you can do the trigonometry and find a tractable way to compute the area contained within the box.