Log fold change calculation errors

[yashd94]

1. guess_is_log in _utils.py does not work correctly, and returns FALSE where our data is in logscale. This results in the is_log1p flag being set incorrectly.

2. _fold_change in _single_cell.py does not account for the is_log1p flag, which results in the fold change function using division instead of subtraction in the case where data is already in logscale. The fix is to pass a corrected is_log1p flag explicitly to the _fold_change function, and toggling between division and subtraction on the basis of this flag.

[noamteyssier]

Hey @yashd94

guess_is_log in _utils.py does not work correctly

The guess_is_log function can sometimes be unstable (it's a guess), we haven't found the best solution for this. If you know your data is log1p a priori I suggest just adding is_log1p=True to the main parallel_differential_expression function call.

However, in your data that is failing the guess_is_log call could you run the following so I can troubleshoot?

mask = np.random.choice(adata.shape[0], size=500, replace=False)
sums = adata[mask].X.sum(axis=1)

print(sums)
print(np.mean(sums)

_fold_change in _single_cell.py does not account for the is_log1p flag

The _fold_change function operates on the means which are assumed to be natural:

pdex/src/pdex/_single_cell.py

Lines 152 to 156 in 1d14b95

	μ_tgt = _sample_mean(x_tgt, is_log1p=is_log1p, exp_post_agg=exp_post_agg)
	μ_ref = _sample_mean(x_ref, is_log1p=is_log1p, exp_post_agg=exp_post_agg)
	fc = _fold_change(μ_tgt, μ_ref, clip_value=clip_value)
	pcc = _percent_change(μ_tgt, μ_ref)

If the data is log1p the _sample_mean functions perform exp1m to correct for it:

pdex/src/pdex/_single_cell.py

Lines 228 to 245 in 1d14b95

	def _sample_mean(
	x: np.ndarray,
	is_log1p: bool,
	exp_post_agg: bool,
	) -> float:
	"""Determine the sample mean of a 1D array.
	Exponenentiates and subtracts one if `is_log1p == True`
	User can decide whether to exponentiate before or after aggregation.
	"""
	if is_log1p:
	if exp_post_agg:
	return np.expm1(np.mean(x))
	else:
	return np.expm1(x).mean()
	else:
	return x.mean()

[yashd94]

@noamteyssier thanks for responding! #2 seems ok, thanks for clarifying. For #1, using the Norman single-cell dataset, here's what I get:

[[1432.0328 ]
[1464.283  ]
[1517.4161 ]
[1352.5338 ]
[1481.2543 ]
[1432.0415 ]
[1415.2266 ]
[1282.4316 ]
[1501.1506 ]
[1353.3912 ]
[1375.6742 ]
[1523.2936 ]
[1482.5549 ]
[1401.6215 ]
[1429.6969 ]
[1311.7258 ]
[1538.4479 ]
[1213.4707 ]
[1426.0156 ]
[1382.3092 ]
[1324.9872 ]
[1576.7528 ]
[1292.3976 ]
[1404.8031 ]
[1611.1306 ]
[1273.3315 ]
[1445.6907 ]
[1315.114  ]
[1363.0419 ]
[1458.781  ]
[1307.4802 ]
[1378.6544 ]
[1412.0355 ]
[1429.4177 ]
[1407.4852 ]
[1262.8561 ]
[1418.1866 ]
[1356.6666 ]
[1414.8905 ]
[1462.0675 ]
[1108.0312 ]
[1379.5458 ]
[1379.6035 ]
[1384.2935 ]
[1388.886  ]
[1437.1759 ]
[1220.7701 ]
[1344.2563 ]
[1395.4008 ]
[1461.3475 ]
[1376.0275 ]
[1384.8578 ]
[1241.8737 ]
[1423.9861 ]
[1589.8868 ]
[1273.4462 ]
[1502.5168 ]
[1435.1167 ]
[1160.5781 ]
[1480.7103 ]
[1111.5791 ]
[1502.4543 ]
[1294.8496 ]
[1431.1622 ]
[1421.6152 ]
[1485.5681 ]
[1206.0719 ]
[1324.0332 ]
[1312.0044 ]
[1506.0643 ]
[1516.2585 ]
[1419.7333 ]
[1402.0576 ]
[1478.2711 ]
[1546.4357 ]
[1461.2205 ]
[1346.3126 ]
[1350.6265 ]
[1462.3326 ]
[1452.6564 ]
[1295.3751 ]
[1472.0425 ]
[1254.2882 ]
[1321.3026 ]
[1461.3632 ]
[1411.8003 ]
[1509.475  ]
[1510.3186 ]
[1419.3245 ]
[1312.2031 ]
[1304.7056 ]
[1358.3435 ]
[1582.7728 ]
[1508.3506 ]
[1277.0978 ]
[1360.019  ]
[1379.015  ]
[1223.8063 ]
[1302.3386 ]
[1587.1556 ]
[1314.314  ]
[1574.4441 ]
[1262.0245 ]
[1424.0624 ]
[1344.2697 ]
[1414.9939 ]
[1459.4314 ]
[1460.5995 ]
[1501.6006 ]
[1315.0823 ]
[1385.9211 ]
[1424.0654 ]
[1434.3469 ]
[1205.7943 ]
[1399.368  ]
[1340.49   ]
[1442.1971 ]
[1494.5436 ]
[1077.2255 ]
[1272.0096 ]
[1186.4266 ]
[1332.8529 ]
[1563.7725 ]
[1395.8765 ]
[1295.7186 ]
[1436.2372 ]
[1449.928  ]
[1450.7451 ]
[1039.804  ]
[1516.3783 ]
[1301.9812 ]
[1511.5671 ]
[1464.0499 ]
[1453.2766 ]
[1121.2954 ]
[1040.741  ]
[1501.6085 ]
[1549.273  ]
[1246.4606 ]
[1447.4095 ]
[1357.1105 ]
[1352.9393 ]
[1357.3164 ]
[1105.8457 ]
[1456.441  ]
[1128.4056 ]
[1430.7943 ]
[1331.3219 ]
[1475.233  ]
[1452.6605 ]
[1403.1742 ]
[1460.8817 ]
[1460.7773 ]
[1512.8617 ]
[1366.6548 ]
[1347.9413 ]
[1395.6096 ]
[1218.4988 ]
[1329.8251 ]
[1364.4717 ]
[1291.1909 ]
[1261.8552 ]
[1476.0961 ]
[1382.551  ]
[1402.3549 ]
[1357.5623 ]
[1322.4324 ]
[1110.5612 ]
[1314.0303 ]
[1340.8353 ]
[1539.6932 ]
[1362.6035 ]
[1456.002  ]
[1143.87   ]
[1317.5387 ]
[1261.7279 ]
[1562.8604 ]
[1303.6028 ]
[1449.4753 ]
[1262.065  ]
[1389.9131 ]
[1470.3855 ]
[1451.8348 ]
[1336.5697 ]
[1481.4459 ]
[1485.5995 ]
[1258.6871 ]
[1327.7761 ]
[1240.2551 ]
[1485.0743 ]
[1276.516  ]
[1389.2196 ]
[1374.8862 ]
[1645.4258 ]
[1390.3718 ]
[1125.2335 ]
[1411.0155 ]
[1513.5726 ]
[1294.8647 ]
[1496.6288 ]
[1044.0461 ]
[1337.7705 ]
[1346.1465 ]
[1022.938  ]
[1244.1006 ]
[1518.717  ]
[1430.1887 ]
[1437.3427 ]
[1187.1239 ]
[1359.3555 ]
[1203.7401 ]
[1337.0763 ]
[1475.424  ]
[1317.5957 ]
[1337.2578 ]
[1439.8141 ]
[1503.5227 ]
[1322.682  ]
[1281.9596 ]
[1444.6488 ]
[1427.2972 ]
[1408.9401 ]
[1307.807  ]
[1354.5676 ]
[1466.2535 ]
[1502.1023 ]
[1332.5208 ]
[1426.9348 ]
[1368.1215 ]
[1135.4017 ]
[1400.5507 ]
[1357.3579 ]
[1304.1421 ]
[1350.9131 ]
[1402.7522 ]
[1440.352  ]
[1303.0103 ]
[1336.9536 ]
[1413.515  ]
[1272.3422 ]
[1274.5336 ]
[1490.195  ]
[1330.7345 ]
[1421.1624 ]
[1475.1556 ]
[1472.3314 ]
[1487.9851 ]
[1429.9226 ]
[1502.573  ]
[1462.4387 ]
[1225.8232 ]
[1478.5027 ]
[1317.2451 ]
[1367.142  ]
[1346.6627 ]
[1449.5381 ]
[1161.1619 ]
[1548.3988 ]
[1291.1461 ]
[1430.9073 ]
[1470.9825 ]
[1141.7451 ]
[1459.9729 ]
[1281.4279 ]
[1222.3739 ]
[1308.0431 ]
[1246.6212 ]
[1305.5759 ]
[1266.0358 ]
[1216.8596 ]
[1413.695  ]
[1317.9866 ]
[1531.9308 ]
[1391.3972 ]
[1445.1343 ]
[1268.2452 ]
[1546.0016 ]
[1369.1431 ]
[1332.2485 ]
[1453.8169 ]
[1382.4432 ]
[1586.6842 ]
[1643.6168 ]
[1351.5881 ]
[1431.3492 ]
[1377.9623 ]
[1493.7328 ]
[1385.1132 ]
[1570.6733 ]
[1329.6525 ]
[1003.98785]
[1165.1934 ]
[1419.4785 ]
[1540.7479 ]
[1339.2897 ]
[1551.121  ]
[1471.1459 ]
[1172.109  ]
[1312.7982 ]
[1432.0924 ]
[1466.7264 ]
[1516.0126 ]
[1341.7185 ]
[1553.162  ]
[1325.6045 ]
[1357.7703 ]
[1416.575  ]
[1539.6298 ]
[1521.5941 ]
[1339.8071 ]
[1275.4799 ]
[1385.2262 ]
[1376.3839 ]
[1480.7782 ]
[1576.3837 ]
[1362.538  ]
[1389.5802 ]
[1496.4104 ]
[1365.1625 ]
[1348.2903 ]
[1408.5759 ]
[1464.2291 ]
[1309.6449 ]
[1396.4109 ]
[1484.6407 ]
[1392.4877 ]
[1460.9786 ]
[1418.8082 ]
[1439.6882 ]
[1441.838  ]
[1400.2993 ]
[1362.2506 ]
[1429.3696 ]
[1585.0195 ]
[1345.6583 ]
[1481.0588 ]
[1310.5215 ]
[1232.4558 ]
[1541.2561 ]
[1565.4899 ]
[1412.5819 ]
[1424.5955 ]
[1360.5774 ]
[1367.3208 ]
[1475.7755 ]
[1387.274  ]
[1515.4536 ]
[1224.1489 ]
[1176.021  ]
[1508.5607 ]
[1395.8358 ]
[1534.6274 ]
[1382.7319 ]
[1410.8547 ]
[1552.3862 ]
[1373.8586 ]
[1422.8579 ]
[1351.9106 ]
[1466.9175 ]
[1535.6118 ]
[ 969.3862 ]
[1466.8217 ]
[1444.6381 ]
[1351.1316 ]
[1519.0149 ]
[1304.7914 ]
[1314.3916 ]
[1295.8243 ]
[1411.2946 ]
[1295.9501 ]
[1563.7118 ]
[1551.1312 ]
[1472.8025 ]
[1377.2028 ]
[1303.5974 ]
[1484.8518 ]
[1371.6338 ]
[1315.2231 ]
[1484.1205 ]
[1355.8345 ]
[1485.4552 ]
[1314.1699 ]
[1396.1406 ]
[1435.9401 ]
[1312.3763 ]
[1335.9049 ]
[1396.848  ]
[1240.2483 ]
[1328.3268 ]
[1403.3151 ]
[1475.8441 ]
[1234.5186 ]
[1408.4683 ]
[1309.6594 ]
[1417.9056 ]
[1444.8967 ]
[1531.7434 ]
[1180.0161 ]
[1452.8046 ]
[1372.058  ]
[1128.294  ]
[1413.6982 ]
[1459.511  ]
[1343.7551 ]
[1448.849  ]
[1331.1981 ]
[1382.9956 ]
[1527.0178 ]
[1371.3893 ]
[1154.7422 ]
[1415.8873 ]
[1387.228  ]
[1444.2028 ]
[1422.247  ]
[1223.9839 ]
[1495.4314 ]
[1221.3256 ]
[1451.869  ]
[1495.2847 ]
[1542.7041 ]
[1582.6931 ]
[1438.6501 ]
[1242.9336 ]
[1339.7932 ]
[1436.6581 ]
[1299.494  ]
[1411.6014 ]
[1568.4637 ]
[1295.8467 ]
[1287.5663 ]
[1348.5537 ]
[1298.2863 ]
[1426.8348 ]
[1517.3793 ]
[1480.5055 ]
[1529.6007 ]
[1531.8671 ]
[1331.388  ]
[1318.7572 ]
[1343.4025 ]
[1475.5015 ]
[1373.8502 ]
[1583.086  ]
[1236.4448 ]
[1351.1493 ]
[1361.4033 ]
[1381.435  ]
[1468.4126 ]
[1335.2489 ]
[1451.0529 ]
[1384.6444 ]
[1373.4404 ]
[1493.1893 ]
[1440.6738 ]
[1394.873  ]
[1304.1885 ]
[1289.7864 ]
[1382.0944 ]
[1464.0408 ]
[1565.1874 ]
[1099.0781 ]
[1328.2393 ]
[1196.5729 ]
[1419.3265 ]
[1307.9686 ]
[1364.5665 ]
[1367.0018 ]
[1417.7294 ]
[1449.2507 ]
[1374.8903 ]
[1228.5175 ]
[1448.7222 ]
[1270.4641 ]
[1420.4623 ]
[1396.4358 ]
[1368.1886 ]
[1057.868  ]
[1344.2449 ]
[1246.6187 ]
[1356.5242 ]
[1176.8563 ]
[1421.7252 ]
[1467.8766 ]
[1335.3436 ]
[1358.1344 ]
[1472.8374 ]
[1513.7299 ]
[1179.3108 ]
[1527.3058 ]
[1399.4908 ]
[1400.9324 ]
[1019.76746]
[1546.3586 ]
[1532.9697 ]
[1282.2954 ]
[1365.5297 ]
[1362.6208 ]
[1514.6222 ]
[1347.221  ]
[1375.2462 ]]
1382.8097