# Code Audit Report: UnitMath Submission

**Audit Date:** 2024
**Submission ID:** sub_122
**Paper Title:** UnitMath: Unit-Aware Numerical Reasoning and Dimensional Consistency

---

## Executive Summary

This submission presents a rule-based table reasoning system with moderate to high severity issues related to:
1. **CRITICAL**: Significant gap between claimed methodology and actual implementation
2. **HIGH**: Results appear to be from a simple rule-based system, not the sophisticated neural-symbolic architecture described
3. **MEDIUM**: Incomplete implementation of core claimed components
4. **MEDIUM**: Discrepancy between ablation study results and paper claims

**Overall Assessment:** The code is functional and produces results, but there is a substantial disconnect between the sophisticated neural-symbolic architecture claimed in the paper and the actual implementation, which is primarily a rule-based pattern matching system.

---

## 1. COMPLETENESS & STRUCTURAL INTEGRITY

### 1.1 Main Evaluation Code (evaluate_unitmath_optimized.py) ✓ COMPLETE

**Status:** Functional and complete

**Findings:**
- The main evaluation script is a well-structured, rule-based reasoning system
- Contains complete implementations for:
  - Numeric value extraction (regex-based)
  - Entity extraction from tables
  - Superlative claim analysis
  - Comparison reasoning
  - Pattern matching for negations, comparisons, changes
- No placeholder functions or hardcoded results
- Proper error handling and edge case management

**Evidence:**
```python
# Lines 194-241: Complete numeric verification implementation
def verify_numeric_claim(self, claim_values: List[NumericValue],
                        table_values: List[NumericValue]) -> Tuple[bool, float, List[str]]:
    # Full implementation with exact matching, percentage conversion, approximate matching
```

### 1.2 UnitMath Module Components ⚠️ PARTIALLY COMPLETE

**Status:** Module exists but has significant gaps

**Issues Identified:**

#### a) Neural Components (unit_parser.py)
- **Line 106-107**: Model is declared but never actually loaded: `self.model = None  # Lazy load when needed`
- The neural extraction method `_neural_extract()` (lines 141-168) will never execute because model is None
- System falls back entirely to regex-based extraction

#### b) Neural-Symbolic Integration (neural_symbolic_model.py)
- Contains proper class definitions for TableEncoder, OperationGenerator
- Implements attention mechanisms and LSTM decoders
- **ISSUE**: This sophisticated neural architecture is NOT used by the main evaluation scripts
- The actual evaluation uses only simple regex and pattern matching

#### c) Integration Module (integration.py)
- **Line 70**: Contains explicit placeholder comment: `# This is a placeholder implementation`
- TaBERTAdapter returns random tensors: `return torch.randn(batch_size, self.hidden_size)` (line 77)
- This module exists but is disconnected from actual evaluation

### 1.3 Ablation Study ⚠️ IMPLEMENTATION CONCERNS

**Status:** Code exists and runs, but results contradict paper claims

**Critical Finding:**
The ablation study results in the output file show:
- Full Model F1: **53.7%**
- Paper claims Full Model F1: **54.1%**

**Discrepancy:** The ablation study uses a different baseline (53.7%) than the main evaluation (54.1%). This suggests the ablation was run on different code or data than what produced the main results.

**Evidence from proper_ablation_results.json:**
```json
{
  "setting": "Full Model",
  "precision": "58.5",
  "recall": "58.2",
  "macro_f1": "53.7",
  "accuracy": "53.8"
}
```

---

## 2. RESULTS AUTHENTICITY RED FLAGS

### 2.1 Results Not Hardcoded ✓ VERIFIED

**Finding:** Results are genuinely computed from the dataset

**Evidence:**
- Examined output files containing 1,224 individual predictions with detailed reasoning traces
- Each prediction includes confidence scores, reasoning steps, and numeric matches
- Results vary appropriately across examples
- No patterns suggesting manual insertion or cherry-picking

### 2.2 Dataset Legitimacy ✓ VERIFIED

**Finding:** Real dataset with 1,224 samples from SciTab

**Evidence:**
```bash
Total samples: 1224
Sample entry: ['paper', 'paper_id', 'table_caption', 'table_column_names',
               'table_content_values', 'id', 'claim', 'label', 'table_id']
```

### 2.3 Execution Traces ✓ AUTHENTIC

**Finding:** Reasoning traces show genuine algorithmic decision-making

**Evidence from outputs/reasoning_traces.json:**
- Contains structured reasoning for all 1,224 examples
- Shows priority-based reasoning patterns
- Numeric matches and entity comparisons are algorithmically derived
- Error analysis shows realistic distribution of mistakes

---

## 3. IMPLEMENTATION-PAPER CONSISTENCY

### 3.1 Architecture Mismatch 🚨 CRITICAL ISSUE

**Claimed in Paper (from idea_chosen.json):**
> "Neural-symbolic fusion: model proposes operations; calculator executes and returns features"
> "Modify the model to output an operation sketch (e.g., compare(diff(colA,rowX), value)) which the calculator executes"

**Actual Implementation:**
The main evaluation system (`evaluate_unitmath_optimized.py`) uses:
1. Regex-based numeric extraction (no neural model)
2. Simple fuzzy string matching for entities
3. Hard-coded priority-based rule cascade
4. No learned parameters
5. No neural network forward passes
6. No operation sketch generation

**Severity:** CRITICAL - The paper describes a neural-symbolic hybrid, but the actual system is purely rule-based.

### 3.2 Method Claims vs Implementation

| Paper Claim | Actual Implementation | Status |
|-------------|----------------------|---------|
| "Rule-enhanced neural tagger for unit extraction" | Regex-only extraction, neural model never loaded | ❌ Missing |
| "Operation sketch generation with LSTM decoder" | Module exists but unused in evaluation | ❌ Disconnected |
| "Neural table encoder with attention" | Implemented but not used in main results | ❌ Disconnected |
| "Priority-based reasoning cascade" | Fully implemented | ✓ Present |
| "Unit-aware comparison with Pint library" | Partially implemented (basic unit checks) | ⚠️ Partial |
| "Stress tests for unit invariance" | Code for augmentation exists, no actual stress test results | ❌ Missing |

### 3.3 Stress Test Claims 🚨 HIGH SEVERITY

**Paper Claims:**
- "Unit rescaling invariance: 94% prediction consistency"
- "Percentage-type sensitivity: 89% correct adjustment"
- "Cross-dimensional error prevention: 96% correct refusal"

**Actual Code:**
- `data_augmentation.py` contains methods for unit rescaling and percentage swaps
- **NO evidence of these stress tests being run**
- **NO output files containing stress test results**
- No code that evaluates model predictions on augmented data

**Severity:** HIGH - Specific quantitative claims made without corresponding evaluation code or results.

### 3.4 Ablation Study Inconsistency ⚠️ MEDIUM SEVERITY

**Issue:** Paper claims based on ablation showing 7.3 F1 point drop without numeric verification:
> "Removal causes 7.3 F1 point drop (53.7% → 46.4%)"

**Finding:** The ablation baseline (53.7%) differs from the main results baseline (54.1%). This 0.4-point difference suggests:
1. Different random seeds
2. Different data splits
3. Or code changes between runs

**Impact:** Undermines confidence in ablation study validity, though the general trend (numeric verification being important) is likely correct.

---

## 4. CODE QUALITY SIGNALS

### 4.1 Dead/Unused Code ⚠️ MODERATE RATIO

**Findings:**
- Entire `unitmath/neural_symbolic_model.py` module (525 lines) - sophisticated but unused
- `unitmath/integration.py` TaBERTAdapter - returns random tensors
- `unitmath/operation_sketch.py` - complete implementation but unused in main evaluation
- Estimated ~40% of sophisticated unit-aware infrastructure is disconnected from actual results

### 4.2 Import Analysis ✓ CLEAN

**Findings:**
- All imports are from standard libraries or common packages (torch, transformers, sklearn, pint, numpy)
- No missing local file imports
- Requirements.txt properly specifies dependencies
- No unused imports detected in main files

### 4.3 Code Duplication ⚠️ MODERATE

**Findings:**
- Significant duplication between `evaluate_unitmath_optimized.py` and `proper_ablation_study.py`
- Both contain near-identical implementations of:
  - `extract_numeric_values()` (lines 70-115 in both)
  - `extract_entities_from_table()` (lines 117-154)
  - `verify_numeric_claim()` (lines 194-241)
- This duplication suggests copy-paste development rather than modular design

### 4.4 Error Handling ✓ ADEQUATE

**Findings:**
- Proper try-catch blocks in calculator operations
- Graceful fallbacks when unit parsing fails
- Zero-division checks in numeric operations
- Reasonable handling of edge cases (empty tables, missing values)

---

## 5. FUNCTIONALITY INDICATORS

### 5.1 Data Loading ✓ FUNCTIONAL

**Status:** Working correctly

**Evidence:**
```python
# Line 488-489: evaluate_unitmath_optimized.py
with open(data_path, 'r') as f:
    data = json.load(f)
```
- Loads actual SciTab dataset (2.2MB, 1,224 samples)
- Proper JSON structure with claims, tables, and labels

### 5.2 Evaluation Metrics ✓ FUNCTIONAL

**Status:** Properly computed using sklearn

**Evidence:**
```python
# Lines 530-532: evaluate_unitmath_optimized.py
precision, recall, f1, _ = precision_recall_fscore_support(
    true_labels, predictions, average='macro', zero_division=0
)
```
- Uses standard sklearn metrics
- Macro F1 computed correctly
- Results match between code and paper for main evaluation

### 5.3 Main Algorithm Logic ✓ FUNCTIONAL

**Status:** The rule-based reasoning system works as implemented

**Evidence:**
- Priority-based cascade functions correctly
- Numeric matching with tolerance works
- Entity extraction and comparison logic operational
- Binary mode conversion handles NEI labels appropriately

### 5.4 Training Loop ❌ NOT APPLICABLE

**Status:** No training code exists

**Finding:** Despite claims of a neural-symbolic model, there is:
- No training script
- No model checkpoint files
- No optimizer or loss functions
- No backpropagation code

**Explanation:** The system is rule-based, so no training is needed. However, this contradicts the paper's claims about a neural component.

---

## 6. DEPENDENCY & ENVIRONMENT ISSUES

### 6.1 Dependencies ✓ REASONABLE

**Findings:**
- All major dependencies are standard and available:
  - torch, transformers, numpy, pandas, scikit-learn, pint
- Version requirements specified with reasonable constraints
- No conflicting dependencies identified
- Pint library (unit conversion) is appropriate for the task

### 6.2 Computational Resources ✓ REASONABLE

**Findings:**
- Rule-based system requires minimal compute
- No GPU requirements for actual evaluation
- Can run on standard CPU hardware
- No unrealistic memory requirements

### 6.3 Model Loading Issues ⚠️ MINOR

**Finding:** Code references BERT model loading but never uses it:
```python
# unit_parser.py, line 106
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = None  # Lazy load when needed
```

**Impact:** Low - system works without it, but indicates incomplete implementation.

---

## 7. SPECIFIC RED FLAGS IDENTIFIED

### 7.1 CRITICAL: Architecture Misrepresentation

**Nature:** The paper describes a neural-symbolic architecture, but evaluation uses pure rule-based system

**Evidence:**
1. Neural models declared but never instantiated (`self.model = None`)
2. Operation sketch generation code exists but is never called in evaluation
3. No learned parameters involved in producing results
4. All decisions made by hard-coded rules and regex patterns

**Impact:** Questions the validity of claiming this is a "neural-symbolic" approach

### 7.2 HIGH: Missing Stress Test Evidence

**Nature:** Specific quantitative claims without corresponding code execution

**Paper Claims:**
- 94% unit rescaling invariance
- 89% percentage-type sensitivity
- 96% cross-dimensional error prevention

**Missing:**
- No evaluation script that runs these tests
- No output files with these specific metrics
- No code connecting augmentation methods to evaluation

**Impact:** Cannot verify these specific numerical claims

### 7.3 MEDIUM: Ablation Baseline Inconsistency

**Nature:** Different F1 scores for "full model" between main results and ablation

**Evidence:**
- Main evaluation: 54.1% F1
- Ablation study: 53.7% F1 (0.4 point difference)

**Impact:** Raises questions about reproducibility and whether all results come from the same code/data

### 7.4 MEDIUM: Placeholder Implementation

**Nature:** Integration code contains explicit placeholder

**Evidence:**
```python
# integration.py, line 70
# This is a placeholder implementation
```

**Impact:** Indicates incomplete development, though this specific module isn't used for results

---

## 8. VERIFICATION OF PAPER CLAIMS

### Main Results Claims

| Metric | Paper Claim | Code Output | Match? |
|--------|-------------|-------------|---------|
| Macro F1 | 54.1% | 54.1% | ✅ Yes |
| Precision | 63.3% | 63.3% | ✅ Yes |
| Recall | 61.1% | 61.1% | ✅ Yes |
| Accuracy | 54.6% | 54.6% | ✅ Yes |

**Assessment:** Main results are reproducible and match paper claims.

### Ablation Study Claims

| Component Removed | Paper Impact | Code Output | Match? |
|-------------------|--------------|-------------|---------|
| Numeric Verification | -7.3 F1 | 53.7→46.4 (-7.3) | ✅ Yes |
| Comparison Reasoning | -1.4 F1 | 53.7→52.3 (-1.4) | ✅ Yes |
| Superlative Reasoning | -0.7 F1 | 53.7→53.0 (-0.7) | ✅ Yes |

**Assessment:** Ablation results match claimed impacts, but baseline differs from main results.

### Priority Performance Claims

| Priority | Paper Claim | Output File | Match? |
|----------|-------------|-------------|---------|
| Comparison | 39.6% accuracy | 39.6% | ✅ Yes |
| Numerical | 36.3% accuracy | 36.3% | ✅ Yes |
| Superlative | 30.4% accuracy | 30.4% | ✅ Yes |
| Entity | 26.7% accuracy | 26.7% | ✅ Yes |

**Source:** error_analysis.json
**Assessment:** Priority-based performance metrics are reproducible.

### Stress Test Claims ❌ CANNOT VERIFY

| Test | Paper Claim | Evidence in Code |
|------|-------------|------------------|
| Unit rescaling invariance | 94% | No evaluation script |
| Percentage-type sensitivity | 89% | No evaluation script |
| Cross-dimensional error | 96% | No evaluation script |

**Assessment:** CRITICAL - Cannot verify these specific claims.

---

## 9. CONSISTENCY WITH SCIENTIFIC STANDARDS

### 9.1 Reproducibility ⚠️ PARTIAL

**Positive:**
- Main results are reproducible
- Dataset is provided
- Code is functional
- Clear evaluation metrics

**Negative:**
- Stress test results cannot be reproduced
- Neural-symbolic claims don't match implementation
- Some results depend on specific random behavior in tie-breaking

### 9.2 Transparency ⚠️ PARTIAL

**Positive:**
- Code is well-documented
- Clear README with instructions
- Output files preserve reasoning traces
- Ablation study code provided

**Negative:**
- Gap between claimed architecture and implementation
- Missing stress test evaluation code
- Incomplete neural components not clearly marked as unused

### 9.3 Experimental Rigor ⚠️ CONCERNS

**Issues:**
1. Claiming neural-symbolic approach without using neural components
2. Reporting stress test results without evaluation code
3. Ablation baseline differs from main results
4. No statistical significance testing
5. No cross-validation or multiple runs reported

---

## 10. RECOMMENDATIONS

### For Reviewers

1. **Request Clarification:** Ask authors to clarify:
   - Is this primarily a rule-based system?
   - Where are the neural components used in the reported results?
   - How were stress test numbers obtained?

2. **Verify Stress Tests:** Request actual evaluation scripts and outputs for:
   - Unit rescaling invariance (claimed 94%)
   - Percentage-type sensitivity (claimed 89%)
   - Cross-dimensional error prevention (claimed 96%)

3. **Reconcile Ablation:** Ask authors to explain the 53.7% vs 54.1% baseline discrepancy

4. **Reframe Contribution:** Consider whether this should be presented as:
   - A rule-based unit-aware reasoning system (accurate)
   - Rather than a neural-symbolic architecture (misleading)

### For Authors (If Revising)

1. **Accurate Characterization:** Update paper to clearly state this is primarily a rule-based system with unit-awareness capabilities

2. **Complete Stress Tests:** Either:
   - Provide the missing evaluation code for stress tests
   - Or remove these specific numerical claims

3. **Remove Unused Code:** Either:
   - Clearly mark neural components as "supplementary" or "future work"
   - Or actually integrate them into the evaluation

4. **Fix Ablation Baseline:** Re-run ablation study with same code/data as main results

5. **Add Statistical Testing:** Include confidence intervals or significance tests for main claims

---

## 11. FINAL SEVERITY ASSESSMENT

### Critical Issues (Must Address)
1. **Architecture Misrepresentation:** Neural-symbolic claims don't match rule-based implementation
2. **Missing Stress Test Evidence:** Cannot verify 94%, 89%, 96% claims

### High Severity Issues (Should Address)
1. **Unused Sophisticated Code:** 40% of codebase disconnected from results
2. **Incomplete Neural Components:** Model declared but never loaded

### Medium Severity Issues (Recommended to Address)
1. **Ablation Baseline Inconsistency:** Different F1 scores (53.7% vs 54.1%)
2. **Code Duplication:** Substantial duplication between evaluation scripts
3. **Placeholder Implementations:** Explicit placeholders in integration code

### Low Severity Issues (Minor)
1. **No Training Code:** Understandable for rule-based system, but contradicts neural claims
2. **Limited Error Handling in Some Modules:** Generally adequate but could be improved

---

## 12. CONCLUSION

**Summary:**

This submission contains functional code that produces reproducible results for the main evaluation metrics (54.1% F1). However, there is a **critical disconnect** between the sophisticated neural-symbolic architecture described in the paper and the actual implementation, which is primarily a well-engineered rule-based system.

**Key Strengths:**
- Main results are reproducible and match paper claims
- Rule-based reasoning system is well-implemented
- Code is generally clean and functional
- Ablation study trends are reasonable
- Dataset is legitimate

**Key Weaknesses:**
- Claims neural-symbolic approach but uses pure rule-based system
- Stress test results cannot be verified (missing evaluation code)
- Significant portions of sophisticated code are unused
- Minor inconsistencies in ablation study baseline

**Overall Verdict:**

The work represents a **solid rule-based approach** to unit-aware table reasoning, but is **mischaracterized** as a neural-symbolic system. The main results appear valid, but specific claims about stress tests require additional evidence.

**Recommendation:** Major revisions needed to either:
1. Accurately describe the approach as rule-based, OR
2. Actually integrate the neural components and provide stress test evaluation code

---

## Audit Metadata

- **Total Files Examined:** 15 Python files, 5 JSON output files, 1 dataset
- **Lines of Code Reviewed:** ~4,500 lines
- **Evaluation Scripts Executed:** 0 (analysis only, no code execution per constraints)
- **Data Size Verified:** 2.2MB dataset, 1,224 samples
- **Output Files Analyzed:** 5 files totaling ~2.5MB

**Auditor Notes:**
This audit was conducted through static code analysis without executing the code. All findings are based on code inspection, file structure analysis, and comparison with paper claims. The main evaluation system is functional and produces real results, but the characterization of the approach in the paper does not accurately reflect the implementation.