Last email discussion

[ppdewolf]

Just as a starter, our email discussion about ground hog day 😄

Dear Peter-Paul, dear Bernhard,

after our discussion about decimals for ptables, we should talk about decimals (maybe once again) of record keys and precision regarding the computation of cellkeys. In this context, we should also discuss about the transformation of cellkeys for the magnitude cases. Up to now, it is intended to be a permutation (i.e., the cellkey of the frequency cell 0.1234 will be changed to 0.2341). Reinhard, Sarah and I already talked about the following issues and want to share them with you.

1. How shall record keys be stored (e.g. if data are delivered to another colleague/angency or exported to a different tool R to SAS ...). Shall they always be stored as character? But then, how many decimals shall be stored? IIRC we have for typical numeric variables (double) 16 places what means 15 to 16 decimal places. Shall we that many decimals places (15 or 16; I also addressed this question to our IT specialists; I shall get a response this week).

2. The precision depends type that can be used by the tool. For example, what if BigDecimal is used (e.g. the Online Service we may use could have this type of format for record keys). Our colleague said, he would still use Bigdecimal but will need the number of decimals we intend to consider. Are you using Bigdecimal in your Java code, Peter-Paul? Is this also relevant for you?

3. Question about the permutation. Bernhard and Reinhard already discussed about that issue when clarifying some testing in R. Bernhard you said, that you would have to switch between variable types (character/numeric) to apply the permutation: a performance issue. A further issue that arises if permutation is carried out: the first decimal place will be put to which decimal place? Is the number of decimals places fixed (see question above)? My alternative suggestion would be, to multiply the cell key by 10 and do another modulo operation. That would be sufficient if we just need a few more cell keys (repeat the approach e.g. for the top_k values). However, what if we would have, as Sarah mentioned, a whole bunch of numerical variables (e.g. income or financial statistics) and each variable needs an individual cell key? Both, permutation and the simple suggestion, would be limited (after 15 or 16 repetitions). Bernhard, you have a more generalized suggestion. Here is the short version of it:
   cellkey2 = { cellkey + cellkey + round(cellkey, 3) } mod 1
   cellkey3 = { cellkey2 + cellkey2 + round(cellkey2, 3) } mod 1 ...
   Could that be applied? The cellkeys (from cellkey to cellkey2) would be dependent similar as cellkeys depend on recordkeys but the new keys are also uniformly distributed. In doing so, we don't have to care about performance (numerical to character and back) or amount of decimal places.

4. When you Reinhard an Bernhard talked about some issues with the testing in R, you have had cell keys with only a few decimals (having only 0.8 permutation, see bullet point 3, permutation did not work correctly). Should there be an input checking, whether record/cellkeys have the amount of decimals we need (see bullet point 1). Or shall these keys with insufficient decimal places be filled up with zeroes? And shall tabulation tools have a definition of output formats to care about the correct number of decimals (see bullet point 1)?

Feel free to correct and answer our thoughts and questions. What are your (additional) ideas to handle the issues.

Many thanks in advance,
Tobias

[ppdewolf]

Dear all,

Just some remarks, comments, thoughts from my side. Not sure if it helps though ;-)

1. In argus we have always taken the "simple" view that ASCII input will always be available. All packages/software we knew about were able to output end input some form of ASCII (free or fixed format). That makes it "easy" for argus: we do not have to change any input or output format whenever some package/software changed its internal (often proprietary) data format.
   When using ASCII input, you need to specify the accompanying metadata. That's why we have the .rda file ;-) One of the things you can/have to specify is the number of digits that is used in the input.
   But:
   1a) Not sure how it is handled when you specify 8 but provide (some) values with less decimals. So, e.g., 0.81927 with 8. As far as I know the compiler I use (but it might be dependent on the used compiler, I guess) will just add zeros, i.e. it will use 0.81927000.
   1b) Not sure what happens when the number of decimals is greater than the maximum precision of the data type. E.g. if 42 and it is converted (cast) to a float.
   1c) This also means that permutations will act like this (see point 3 below for the code)
   with 8: 0.1234 -> 0.23400001 (well, at least with the GCC compiler that adds zeros)
   with 4: 0.1234 -> 0.2341

2. Since parts of tau-argus are written in Java, parts in C/C++, parts in Fortran, parts in Pascal, we have to exchange data between programming languages all the time. However, I am not too familiar with all languages and thus not with the differences in precisions. E.g., an Integer in Java is comparable to a long in C/C++. Not sure how this may interact.

3. For the "transformation" of 0.897 to 0.978 I also use a string representation of the number, apply the transformation and then convert back to a number. I don't think the performance issue is too big though. For this transformation, I use the that were defined in the rda-file. That is, for the cell keys I use the same number of decimals as was specified for the record keys and do the permutation over that number of decimals (nDec in below quoted code).
   My code in C/C++ (likely not being efficient, but it works):
   Double TauArgusJava::ShiftFirstDigit(double key, int nDec){
   char buffer[64];
   ASSERT(nDec < 62);
   snprintf(buffer, 64, "%*.*f", nDec+2, nDec, key);
   std::string cstr(buffer);
   cstr = "0." + cstr.substr(3) + cstr.substr(2,1);
   return atof(cstr.c_str());
   }

4. The ptable file I read "as is". I.e., there is no metadata description and I just read each entry and convert it to a double.

5. I guess that at least the following should hold: the number of decimals in the record keys should be less or equal to the number of decimals in the probabilities as given in the ptable file. When the number of digits of a cell key is more than the probabilities in the ptable have, you cannot make a distinction between some seemingly (and perhaps even intended?) different cell keys close to a pij_ub. When the number of digits of a cell key is less than the probabilities in the ptable have it seems to be no problem?

6. Do you really need many decimal places? What would be the problem if some records have the same record key? Or some cells have the same cell key?

7. As I understood, the permutation is applied to the cell key when you have more than one contributor to deal with (TopK > 1). But I assume that the TopK will never be too big? Actually I restricted it in argus to <= 5 (for the sake of the GUI). So I do not get the "However, what if we would have, as Sarah mentioned, a whole bunch of numerical variables (e.g. income or financial statistics) and each variable needs an individual cell key?". Shouldn't you use different record keys to start with, if you really need different cell keys for different variables? Why use the permutation trick in that case?

Summarizing: I see a lot of advantages in specifying the number of digits. But there are still some "issues" dealing with different compilers as well.

And I am just wondering whether the differences will be that big anyway, when you deal with say 12 decimals and the different implementations only differ from the thirteenth decimal place...

Peter-Paul

[ppdewolf]

hi everybody,

strangely, I did not receive Tobias original mail, only Peter-Pauls reply. Attached my 2 cents;

1b: i have solved it now the same way, if less than the number of specified digits is available, I just add zeros
1c: same in my R implementation
2: i think we will never achieve complete and full identical outputs but as PP mentioned in his final comment, I also think these differences are edge-cases that we may just decide not to care about
3: i agree that the overhead of converting to/from strings and numbers is small. perhaps we should leave things as they are :)
5: @pp: did you implement this as a "hard check" already?
6: perhaps in large tables with lot of single units in the cells we get a custom perturbation pattern if the record-key (=cell key) for these cells does not spread out too much? but other than that, I do not see any issue with small(er) number of digits in rkeys
7: I've also limited topK (to <= 6) I think; as for individual cell keys by variable; if required; why do not compute original cell keys and modify them using a function that has a seed based on the values itself? e.g mean, median, ...? then you would still be able to get reproducable cellkeys by variable while still having to store only a single rkey/ckey variable and can compute the final keys it on-the-fly.

Greets and have a safe and (happy) Easter, Bernhard

[ppdewolf]

Dear Peter-Paul, dear Bernhard,

thanks. You will get a response next week (especially to 5 and 6 with an example why there could be needed more decimals).

Have nice Easter as well,
Tobias