Request of clarification over some mathematical definitions

[enricoteofilo]

Hello.
Sorry to bother, but I was looking for explicit mathematical definitions in the documentation and I haven' t found all I was looking for. Hence I'm now asking for your support to get confirmation on some mathematical definitions.

First, can you please confirm that, even if weights1/weights2 are passed to the pair-counting functions (like Corrfunc.theory.DD or Corrfunc.mocks.DDrppi_mocks), the pair counts per-bin are not affected by them? If that's the case, as I've understood for the documentation, is the pair counts in the $n-$th bin given by the following expression?

$$DD(n) = \sum_{i\neq j}\Theta^{ij}_{n}$$

where the $\Theta$ gives 1 if the pair $i,j$ is in the $n-$th bin and 0 otherwise:

$$\Theta^{ij}_{n} = \int_{\Omega_n}d^{3}\vec{r}\delta(r-|\vec{x}_i-\vec{x}_j|)\delta(\mu-\mu_{ij})\delta(\phi-\phi_{ij})$$

I expect the $n-$th bin (that I've called $\Omega_n$) to be defined, for example for the DDsmu_mocks, as:

$$\Omega_n = [r_n-\Delta r/2,r_n+\Delta r/2]\times[\mu_n-\Delta\mu/2,\mu_n+\Delta\mu/2]\times[0,2\pi]$$

Is this the actual definition of the bin internal to the function?
If all stated above holds, can you also confirm that, although the pair counts $DD,DR,RR$ are not influenced by the optional weights, the weightavg column of the results gives the average pair weight per bin and that the definition of this average is the following?

$$w_{avg}(n) = \frac{1}{DD(n)} \sum_{i\neq j}\Theta^{ij}_{n}w_{i}w_{j}$$

I may have other inquiries on the mathematical definitions of the quantities implemented in the code. I will keep on updating this first post and reply to the thread if that's the case and IF that's ok

[lgarrison]

First, can you please confirm that, even if weights1/weights2 are passed to the pair-counting functions (like Corrfunc.theory.DD or Corrfunc.mocks.DDrppi_mocks), the pair counts per-bin are not affected by them?

Yes, that's right.

If that's the case, as I've understood for the documentation, is the pair counts in the n − th bin given by the following expression?
D D ( n ) = ∑ i ≠ j Θ n i j

Not quite; if a pair has zero separation, and the user provides a binning that includes zero separation, than that pair is counted in the appropriate bin. This is for symmetry between auto- and cross-correlations. In other words, for autocorrelations, the sum doesn't exclude the $i=j$ case.

I expect the n − th bin (that I've called Ω n ) to be defined, for example for the DDsmu_mocks, as:
Ω n = [ r n − Δ r / 2 , r n + Δ r / 2 ] × [ μ n − Δ μ / 2 , μ n + Δ μ / 2 ] × [ 0 , 2 π ]

All of the correlation functions accept a user-defined array of radial bin edges, which are inclusive on the lower bound and exclusive on the upper. Note that the user passes bin edges, not bin centers. Likewise, the mu binning uses nmu_bins from 0 to mu_max by internally using nmu_bins + 1 bin edges.

If all stated above holds, can you also confirm that, although the pair counts D D , D R , R R are not influenced by the optional weights, the weightavg column of the results gives the average pair weight per bin and that the definition of this average is the following?
w a v g ( n ) = 1 D D ( n ) ∑ i ≠ j Θ n i j w i w j

This depends on the weight_type specified by the user. But in the default case of 'pair_product', this is correct, with the $i=j$ caveat from above.

[manodeep]

Thanks @enricoteofilo for asking for clarification. What @lgarrison said is exactly correct (especially the caveat - which isn't really a surprise since he was the one that flagged that edge-case about Rmin==0 and implemented the fix :))

@lgarrison Should we copy-paste your explanation into a "Definitions" docs page?

[lgarrison]

Sure! I'm not sure the docs are building right now, though.

[enricoteofilo]

If that's the case, as I've understood for the documentation, is the pair counts in the n − th bin given by the following expression?
D D ( n ) = ∑ i ≠ j Θ n i j

Not quite; if a pair has zero separation, and the user provides a binning that includes zero separation, than that pair is counted in the appropriate bin. This is for symmetry between auto- and cross-correlations. In other words, for autocorrelations, the sum doesn't exclude the i = j case.

Thanks for the confirmation. I had caught that before and I had already put an additional line in my Python code so that $DD(r=0,\mu=0)$ is replaced by $DD(r=0,\mu=0)-N_{obs}$. This is valid for all the pair-counting functions, either in the theory section of mocks section, right?
Anyway, bins not containing $r=0$ and $\mu=0$ should not be influenced by this caveat, so that's not a huge problem unless you are trying to actually normalize the pair-counts per bin. Also does not create issues with $DR$.
Anwyay, I would suggest adding a kwarg named self_term such that self_term=False corresponds to requiring $i\neq j$. The default value should be self_term=True for compatibility with existing code. The npairs and weightavg fields would need to be adjusted accordingly when set to False.

I expect the n − th bin (that I've called Ω n ) to be defined, for example for the DDsmu_mocks, as:
Ω n = [ r n − Δ r / 2 , r n + Δ r / 2 ] × [ μ n − Δ μ / 2 , μ n + Δ μ / 2 ] × [ 0 , 2 π ]

All of the correlation functions accept a user-defined array of radial bin edges, which are inclusive on the lower bound and exclusive on the upper. Note that the user passes bin edges, not bin centers. Likewise, the mu binning uses nmu_bins from 0 to mu_max by internally using nmu_bins + 1 bin edges.

Thank you very much. This was clear from the documentation but I had reported here the definition that would correspond to someone plotting $\xi(r)$ on the $y-$axis against the centerbins $r_n$ on the $x-$ axis. The definition of the bins more coherent with the actual implementation of the bins in, for example, DDsmu_mocks, I think would be:

$$\Omega_{a}^{c} = [r_a,r_{a+1})\times[\mu_b,\mu_{b+1})\times[0,2\pi]$$

(I've used the same indices nomenclature that is used in RascalC papers from the DESI team).

If all stated above holds, can you also confirm that, although the pair counts D D , D R , R R are not influenced by the optional weights, the weightavg column of the results gives the average pair weight per bin and that the definition of this average is the following?
w a v g ( n ) = 1 D D ( n ) ∑ i ≠ j Θ n i j w i w j

This depends on the weight_type specified by the user. But in the default case of 'pair_product', this is correct, with the i = j caveat from above.

OK, thanks, then the weighted-pair counts would be calculated with results["npairs"]*results["weightavg"] and this would be equivalent to:

$$DD_{a}^{c} = \sum_{i,j}\Theta_{a}^{c}(i,j)w_{i}w_{j}$$