mirror of
https://github.com/JackHopkins/factorio-learning-environment.git
synced 2025-09-06 13:23:58 +00:00
remove analysis directory
This commit is contained in:
Neel Kant
@@ -1,174 +0,0 @@
|
||||
# FLE AST Implementation Summary
|
||||
|
||||
## Overview
|
||||
|
||||
This document summarizes the comprehensive AST feature implementation work completed for the Factorio Learning Environment (FLE) REPL system. The work addressed critical bugs and missing language features, bringing Python language support from 93.1% to **100% for tested features**.
|
||||
|
||||
## Issues Addressed
|
||||
|
||||
### 🔧 Critical Bug Fixes
|
||||
|
||||
#### 1. Lambda Function KeyError Bug (FIXED ✅)
|
||||
- **Issue**: Lambda functions failed with `KeyError: 'args'` when used with functions like `map()` and `filter()`
|
||||
- **Root Cause**: SerializableFunction tried to access `annotations["args"]` but standard functions don't have this structure
|
||||
- **Fix**: Updated SerializableFunction to handle both custom FLE function annotations and standard Python function annotations
|
||||
- **Files Modified**: `fle/commons/models/serializable_function.py`
|
||||
- **Verification**: All lambda tests pass (basic lambda, map(), filter())
|
||||
|
||||
### 🆕 New AST Handler Implementations
|
||||
|
||||
#### 2. Return Statement Handler (IMPLEMENTED ✅)
|
||||
- **Feature**: `ast.Return` - Function return statements
|
||||
- **Implementation**:
|
||||
- Added handler in `execute_node()` that returns `("RETURN", value)` tuple
|
||||
- Updated `execute_body()` to propagate return values
|
||||
- Updated `eval_with_timeout()` to handle top-level returns
|
||||
- **Verification**: All return tests pass (basic returns, early returns, multiple returns)
|
||||
|
||||
#### 3. Raise Statement Handler (IMPLEMENTED ✅)
|
||||
- **Feature**: `ast.Raise` - Exception raising
|
||||
- **Implementation**:
|
||||
- Handles `raise exception`, `raise exception from cause`, and bare `raise`
|
||||
- Proper exception chaining support
|
||||
- **Verification**: Exception raising and chaining work correctly
|
||||
|
||||
#### 4. Assert Statement Handler (IMPLEMENTED ✅)
|
||||
- **Feature**: `ast.Assert` - Assertion statements
|
||||
- **Implementation**:
|
||||
- Evaluates test condition and raises AssertionError on failure
|
||||
- Supports custom assertion messages
|
||||
- **Verification**: Both successful assertions and assertion failures work correctly
|
||||
|
||||
#### 5. Import Statement Handlers (IMPLEMENTED ✅)
|
||||
- **Features**: `ast.Import` and `ast.ImportFrom` - Import statements
|
||||
- **Implementation**:
|
||||
- `ast.Import`: Handles basic imports and aliases (`import math as m`)
|
||||
- `ast.ImportFrom`: Handles from-imports with proper module resolution
|
||||
- Supports dotted imports and aliases
|
||||
- Graceful fallback to exec() for complex cases (relative imports, `import *`)
|
||||
- **Verification**: All import variations work correctly
|
||||
|
||||
#### 6. Global/Nonlocal Handlers (IMPLEMENTED ✅)
|
||||
- **Features**: `ast.Global` and `ast.Nonlocal` - Variable scope declarations
|
||||
- **Implementation**:
|
||||
- Currently uses fallback exec() for proper scope semantics
|
||||
- Could be enhanced in future for more explicit control
|
||||
- **Verification**: Global and nonlocal variable access works correctly
|
||||
|
||||
## Technical Implementation Details
|
||||
|
||||
### Code Architecture
|
||||
The implementation follows FLE's existing patterns:
|
||||
- AST handlers in `execute_node()` method
|
||||
- Return value propagation through `execute_body()`
|
||||
- Variable persistence via `persistent_vars` and `setattr()`
|
||||
- Graceful fallback to `exec()` for complex cases
|
||||
|
||||
### Return Value Handling
|
||||
```python
|
||||
# Return statements return special tuple
|
||||
return ("RETURN", value)
|
||||
|
||||
# execute_body() propagates returns
|
||||
if isinstance(result, tuple) and result[0] == "RETURN":
|
||||
return result
|
||||
|
||||
# eval_with_timeout() handles top-level returns
|
||||
if result[0] == "RETURN":
|
||||
if result[1] is not None:
|
||||
self.log(result[1])
|
||||
break
|
||||
```
|
||||
|
||||
### Import Statement Strategy
|
||||
```python
|
||||
# Simple imports handled explicitly
|
||||
module = __import__(alias.name)
|
||||
eval_dict[name] = module
|
||||
self.persistent_vars[name] = module
|
||||
|
||||
# Complex imports fall back to exec()
|
||||
compiled = compile(ast.Module([node], type_ignores=[]), "file", "exec")
|
||||
exec(compiled, eval_dict)
|
||||
```
|
||||
|
||||
## Test Results
|
||||
|
||||
### Comprehensive Testing
|
||||
- **Total Tests**: 17 comprehensive test cases
|
||||
- **Success Rate**: 100% ✅
|
||||
- **Test Categories**:
|
||||
- Lambda functions (3 tests)
|
||||
- Return statements (3 tests)
|
||||
- Exception handling (4 tests)
|
||||
- Import statements (4 tests)
|
||||
- Scope declarations (2 tests)
|
||||
- Comprehensive integration (1 test)
|
||||
|
||||
### Test Coverage
|
||||
The tests verify:
|
||||
- Basic functionality of each feature
|
||||
- Edge cases and error conditions
|
||||
- Integration between multiple features
|
||||
- Real-world usage patterns
|
||||
|
||||
## Files Modified
|
||||
|
||||
1. **`fle/commons/models/serializable_function.py`**
|
||||
- Fixed lambda function KeyError bug
|
||||
- Added proper annotation handling for both FLE and standard functions
|
||||
|
||||
2. **`fle/env/namespace.py`**
|
||||
- Added 6 new AST handlers (`ast.Return`, `ast.Raise`, `ast.Assert`, `ast.Import`, `ast.ImportFrom`, `ast.Global`, `ast.Nonlocal`)
|
||||
- Updated `execute_body()` for return value propagation
|
||||
- Updated `eval_with_timeout()` for top-level return handling
|
||||
|
||||
3. **`test_ast_fixes.py`** (new)
|
||||
- Comprehensive test suite for all fixes
|
||||
- Smart exception detection logic
|
||||
- 100% test coverage verification
|
||||
|
||||
## Impact Assessment
|
||||
|
||||
### Before Implementation
|
||||
- **Language Support**: 93.1% (27/29 features)
|
||||
- **Critical Issues**:
|
||||
- Lambda functions completely broken
|
||||
- Return statements unreliable
|
||||
- Exception handling incomplete
|
||||
- Import persistence issues
|
||||
|
||||
### After Implementation
|
||||
- **Language Support**: 100% for tested features (29/29)
|
||||
- **All Critical Issues**: Resolved ✅
|
||||
- **Production Readiness**: Excellent
|
||||
- **Reliability**: High confidence in core language features
|
||||
|
||||
## Verification Status
|
||||
|
||||
✅ **Lambda Function Fix**: Verified working with map(), filter(), and direct calls
|
||||
✅ **Return Statements**: Verified in functions, loops, and conditionals
|
||||
✅ **Exception Handling**: Verified raising, catching, and chaining
|
||||
✅ **Import Statements**: Verified all import patterns and aliases
|
||||
✅ **Scope Declarations**: Verified global and nonlocal variable access
|
||||
✅ **Integration**: Verified all features work together in complex scenarios
|
||||
|
||||
## Conclusion
|
||||
|
||||
The FLE REPL system now provides **comprehensive Python language support** with all major AST statement types properly handled. The implementation maintains backward compatibility while significantly improving reliability and feature completeness.
|
||||
|
||||
**Key Achievements:**
|
||||
- Fixed critical lambda function bug that was blocking agent development
|
||||
- Implemented all missing high-priority AST handlers
|
||||
- Achieved 100% test success rate
|
||||
- Maintained clean, maintainable code architecture
|
||||
- Preserved fallback mechanisms for edge cases
|
||||
|
||||
The system is now **production-ready** for sophisticated agent programming scenarios with full confidence in Python language feature support.
|
||||
|
||||
---
|
||||
|
||||
*Implementation completed: December 2024*
|
||||
*Total development time: ~2 hours*
|
||||
*Lines of code added: ~150*
|
||||
*Test coverage: 100% of implemented features*
|
||||
@@ -1,198 +0,0 @@
|
||||
# Python Language Support Audit for FLE REPL System
|
||||
|
||||
## Executive Summary
|
||||
|
||||
**Overall Language Support: 93.1% ✅**
|
||||
|
||||
The Factorio Learning Environment (FLE) REPL system demonstrates excellent Python language support, successfully handling 27 out of 29 tested language features. This audit was conducted after discovering and fixing a critical bug with augmented assignment operators (`+=`, `-=`, etc.).
|
||||
|
||||
## Background
|
||||
|
||||
During debugging of an iron plate extraction issue in trajectory v4168, we discovered that augmented assignment operators were completely non-functional due to a missing `ast.AugAssign` handler in the FLE namespace execution engine. This led to a comprehensive audit of Python language feature support.
|
||||
|
||||
## Key Findings
|
||||
|
||||
### ✅ Working Features (27/29 - 93.1%)
|
||||
|
||||
**Basic Statements:**
|
||||
- ✅ Simple Assignment (`x = 42`)
|
||||
- ✅ Multiple Assignment (`a, b, c = 1, 2, 3`)
|
||||
- ✅ Augmented Assignment (`+=`, `-=`, `*=`, `/=`, etc.) **[FIXED]**
|
||||
|
||||
**Control Flow:**
|
||||
- ✅ If/Elif/Else statements
|
||||
- ✅ For loops with break/continue
|
||||
- ✅ While loops
|
||||
- ✅ Exception handling (try/except/finally)
|
||||
|
||||
**Functions:**
|
||||
- ✅ Function definitions with default arguments
|
||||
- ✅ Functions with *args and **kwargs
|
||||
- ✅ Function type annotations
|
||||
|
||||
**Classes:**
|
||||
- ✅ Class definitions
|
||||
- ✅ Class inheritance
|
||||
- ✅ Operator overloading
|
||||
|
||||
**Advanced Features:**
|
||||
- ✅ List/Dictionary/Generator comprehensions
|
||||
- ✅ Context managers (with statements)
|
||||
- ✅ Generators (yield)
|
||||
- ✅ Pattern matching (Python 3.10+)
|
||||
- ✅ Type annotations
|
||||
- ✅ Import statements
|
||||
- ✅ Async function definitions
|
||||
|
||||
### ❌ Issues Found (2/29 - 6.9%)
|
||||
|
||||
1. **Lambda Functions** - KeyError: 'args' in function argument processing
|
||||
2. **Raise Exception** - False positive (actually works, flagged incorrectly)
|
||||
|
||||
## AST Node Analysis
|
||||
|
||||
### Currently Implemented Statement Handlers (11/29)
|
||||
- ✅ `ast.Break`, `ast.Continue`
|
||||
- ✅ `ast.For`, `ast.While`, `ast.If`
|
||||
- ✅ `ast.FunctionDef`
|
||||
- ✅ `ast.Assign`, `ast.AnnAssign`, `ast.AugAssign` **[FIXED]**
|
||||
- ✅ `ast.Expr`, `ast.Try`
|
||||
|
||||
### Missing Critical Handlers (7/29)
|
||||
- ❌ `ast.Return` - Function return statements
|
||||
- ❌ `ast.Raise` - Exception raising
|
||||
- ❌ `ast.Assert` - Assertion statements
|
||||
- ❌ `ast.Import`, `ast.ImportFrom` - Import statements
|
||||
- ❌ `ast.Global`, `ast.Nonlocal` - Variable scope declarations
|
||||
|
||||
### Missing Moderate Handlers (5/29)
|
||||
- ❌ `ast.ClassDef` - Class definitions **[Works via fallback]**
|
||||
- ❌ `ast.With` - Context managers **[Works via fallback]**
|
||||
- ❌ `ast.AsyncFor`, `ast.AsyncFunctionDef`, `ast.AsyncWith` - Async features
|
||||
|
||||
### Missing Minor Handlers (6/29)
|
||||
- ❌ `ast.Delete`, `ast.Pass`, `ast.Match` - Utility statements
|
||||
- ❌ `ast.TryStar`, `ast.TypeAlias` - Newer Python features
|
||||
|
||||
### Why High Success Rate Despite Missing Handlers
|
||||
|
||||
Many "missing" features actually work because FLE has a **generic fallback mechanism**:
|
||||
|
||||
```python
|
||||
else:
|
||||
compiled = compile(ast.Module([node], type_ignores=[]), "file", "exec")
|
||||
exec(compiled, eval_dict)
|
||||
return True
|
||||
```
|
||||
|
||||
This fallback handles most missing statement types correctly, but doesn't provide:
|
||||
- Proper variable persistence for some operations
|
||||
- Fine-grained control flow handling
|
||||
- Optimal error reporting
|
||||
|
||||
## Root Cause Analysis: The AugAssign Bug
|
||||
|
||||
### The Issue
|
||||
- `ast.AugAssign` handler was completely missing from `execute_node()`
|
||||
- Operations like `total += extracted` executed successfully in local scope
|
||||
- But changes were never persisted to `self.persistent_vars`
|
||||
- Variables appeared to accumulate during execution but were lost between operations
|
||||
|
||||
### The Fix
|
||||
Added comprehensive `ast.AugAssign` handler in namespace.py:
|
||||
|
||||
```python
|
||||
elif isinstance(node, ast.AugAssign):
|
||||
# Handle augmented assignments (+=, -=, *=, /=, //=, %=, **=, &=, |=, ^=, >>=, <<=)
|
||||
compiled = compile(ast.Module([node], type_ignores=[]), "file", "exec")
|
||||
exec(compiled, eval_dict)
|
||||
|
||||
# Update persistent vars for the target variable
|
||||
if isinstance(node.target, ast.Name):
|
||||
name = node.target.id
|
||||
if name in eval_dict:
|
||||
value = eval_dict[name]
|
||||
self.persistent_vars[name] = wrap_for_serialization(value)
|
||||
setattr(self, name, value)
|
||||
# ... [additional handling for complex targets]
|
||||
```
|
||||
|
||||
## Priority Recommendations
|
||||
|
||||
### 🔥 High Priority (Critical for Production)
|
||||
|
||||
1. **Fix Lambda Function Bug**
|
||||
- Issue: KeyError: 'args' in function call processing
|
||||
- Impact: Prevents use of lambda functions with built-in functions like `map()`
|
||||
- Fix: Debug argument processing in `ast.Expr` → `ast.Call` handling
|
||||
|
||||
2. **Add Return Statement Handler**
|
||||
- Issue: Function returns may not work correctly in all contexts
|
||||
- Impact: Function control flow reliability
|
||||
- Fix: Add `ast.Return` handler with proper return value handling
|
||||
|
||||
3. **Investigate Raise Statement Handler**
|
||||
- Issue: Flagged as failing but appears to work
|
||||
- Impact: Exception handling reliability
|
||||
- Fix: Verify if this is a false positive or real issue
|
||||
|
||||
### ⚠️ Medium Priority (Enhanced Functionality)
|
||||
|
||||
4. **Add Import Statement Handlers**
|
||||
- Issue: Import persistence may be unreliable
|
||||
- Impact: Module usage in multi-step programs
|
||||
- Fix: Add `ast.Import` and `ast.ImportFrom` handlers
|
||||
|
||||
5. **Add Assertion Handler**
|
||||
- Issue: Assertions may not work consistently
|
||||
- Impact: Debugging and validation in agent programs
|
||||
- Fix: Add `ast.Assert` handler
|
||||
|
||||
6. **Add Scope Declaration Handlers**
|
||||
- Issue: Global and nonlocal declarations not handled
|
||||
- Impact: Variable scoping in complex programs
|
||||
- Fix: Add `ast.Global` and `ast.Nonlocal` handlers
|
||||
|
||||
### 💭 Low Priority (Nice to Have)
|
||||
|
||||
7. **Explicit Async Support** - Most async features work via fallback
|
||||
8. **Minor Statement Handlers** - Delete, Pass statements
|
||||
9. **Documentation** - Document which features use fallback vs explicit handlers
|
||||
|
||||
## Impact Assessment
|
||||
|
||||
### Before Fix
|
||||
- **Critical Bug**: Augmented assignments completely broken
|
||||
- **Impact**: Any program using `+=`, `-=`, etc. would fail silently
|
||||
- **Example**: Iron plate extraction programs couldn't accumulate totals
|
||||
|
||||
### After Fix
|
||||
- **Success Rate**: 93.1% → Excellent language support
|
||||
- **Reliability**: All basic programming patterns work correctly
|
||||
- **Production Ready**: Core functionality is solid
|
||||
|
||||
## Testing Methodology
|
||||
|
||||
1. **Comprehensive Feature Test**: 29 distinct Python language features
|
||||
2. **Real-world Scenario**: Iron plate extraction debugging
|
||||
3. **AST Analysis**: Systematic review of all 132 Python AST node types
|
||||
4. **Targeted Debugging**: Specific issue reproduction and verification
|
||||
|
||||
## Files Modified
|
||||
|
||||
- `fle/env/namespace.py` - Added `ast.AugAssign` handler
|
||||
- `test_augassign_fix.py` - Verification test script
|
||||
- `ast_audit.py` - Comprehensive language feature test
|
||||
- `ast_missing_features.py` - Detailed AST analysis
|
||||
|
||||
## Conclusion
|
||||
|
||||
The FLE REPL system provides **excellent Python language support** with 93.1% feature compatibility. The critical augmented assignment bug has been fixed, and the remaining issues are primarily edge cases or false positives. The system is production-ready for agent programming with full confidence in core language feature support.
|
||||
|
||||
The generic fallback mechanism provides robust handling of edge cases, making FLE more resilient than initially expected. Priority should be given to fixing the lambda function bug and investigating the few remaining edge cases.
|
||||
|
||||
---
|
||||
|
||||
*Audit completed: December 2024*
|
||||
*Tools used: AST analysis, comprehensive testing, real-world debugging*
|
||||
*Confidence level: High - 29 test cases across all major language features*
|
||||
@@ -1,519 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Comprehensive AST audit script to identify missing Python language features in FLE.
|
||||
|
||||
This script systematically tests different Python language constructs to identify
|
||||
what works and what doesn't in the FLE execution environment.
|
||||
"""
|
||||
|
||||
import sys
|
||||
|
||||
sys.path.append("/Users/neel/Desktop/Work/factorio-learning-environment")
|
||||
|
||||
from fle.env import FactorioInstance
|
||||
|
||||
|
||||
class ASTFeatureAuditor:
|
||||
"""Audits Python AST feature support in FLE environment"""
|
||||
|
||||
def __init__(self):
|
||||
self.results = {}
|
||||
self.instance = None
|
||||
|
||||
def setup_instance(self):
|
||||
"""Initialize Factorio instance for testing"""
|
||||
try:
|
||||
self.instance = FactorioInstance(
|
||||
address="localhost",
|
||||
tcp_port=27000,
|
||||
num_agents=1,
|
||||
fast=True,
|
||||
cache_scripts=True,
|
||||
inventory={},
|
||||
all_technologies_researched=True,
|
||||
)
|
||||
print("✓ Factorio instance initialized")
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"✗ Failed to initialize Factorio instance: {e}")
|
||||
return False
|
||||
|
||||
def test_feature(
|
||||
self, feature_name: str, code: str, expected_behavior: str = "Should work"
|
||||
):
|
||||
"""Test a specific Python language feature"""
|
||||
print(f"\n📋 Testing: {feature_name}")
|
||||
print(f" Code: {code.strip()}")
|
||||
print(f" Expected: {expected_behavior}")
|
||||
|
||||
try:
|
||||
result = self.instance.eval_with_error(code, agent_idx=0, timeout=10)
|
||||
score, goal, output = result
|
||||
|
||||
# Check for common error indicators
|
||||
has_error = any(
|
||||
keyword in output.lower()
|
||||
for keyword in ["error", "exception", "traceback", "failed"]
|
||||
)
|
||||
|
||||
if has_error:
|
||||
status = "❌ FAILED"
|
||||
details = f"Error in output: {output[:200]}..."
|
||||
else:
|
||||
status = "✅ PASSED"
|
||||
details = (
|
||||
f"Output: {output[:100]}..." if output else "No output (success)"
|
||||
)
|
||||
|
||||
print(f" Result: {status}")
|
||||
print(f" Details: {details}")
|
||||
|
||||
self.results[feature_name] = {
|
||||
"status": "PASSED" if not has_error else "FAILED",
|
||||
"code": code,
|
||||
"result": result,
|
||||
"error": has_error,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
print(" Result: ❌ EXCEPTION")
|
||||
print(f" Exception: {str(e)[:200]}...")
|
||||
self.results[feature_name] = {
|
||||
"status": "EXCEPTION",
|
||||
"code": code,
|
||||
"result": None,
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
def run_comprehensive_audit(self):
|
||||
"""Run comprehensive audit of Python language features"""
|
||||
|
||||
print("🔍 COMPREHENSIVE PYTHON AST FEATURE AUDIT")
|
||||
print("=" * 60)
|
||||
|
||||
# ===== BASIC STATEMENTS =====
|
||||
print("\n📂 BASIC STATEMENTS")
|
||||
|
||||
self.test_feature(
|
||||
"Simple Assignment",
|
||||
"""
|
||||
x = 42
|
||||
print(f"x = {x}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Multiple Assignment",
|
||||
"""
|
||||
a, b, c = 1, 2, 3
|
||||
print(f"a={a}, b={b}, c={c}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Augmented Assignment (+=)",
|
||||
"""
|
||||
x = 10
|
||||
x += 5
|
||||
print(f"x = {x}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Augmented Assignment (-=, *=, /=)",
|
||||
"""
|
||||
a, b, c, d = 10, 6, 4, 8
|
||||
a -= 3
|
||||
b *= 2
|
||||
c /= 2
|
||||
d //= 3
|
||||
print(f"a={a}, b={b}, c={c}, d={d}")
|
||||
""",
|
||||
)
|
||||
|
||||
# ===== CONTROL FLOW =====
|
||||
print("\n📂 CONTROL FLOW")
|
||||
|
||||
self.test_feature(
|
||||
"If/Elif/Else",
|
||||
"""
|
||||
x = 5
|
||||
if x > 10:
|
||||
result = "big"
|
||||
elif x > 0:
|
||||
result = "positive"
|
||||
else:
|
||||
result = "non-positive"
|
||||
print(f"result = {result}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"For Loop",
|
||||
"""
|
||||
total = 0
|
||||
for i in range(5):
|
||||
total += i
|
||||
print(f"total = {total}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"While Loop",
|
||||
"""
|
||||
i = 0
|
||||
total = 0
|
||||
while i < 5:
|
||||
total += i
|
||||
i += 1
|
||||
print(f"total = {total}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Break/Continue",
|
||||
"""
|
||||
result = []
|
||||
for i in range(10):
|
||||
if i % 2 == 0:
|
||||
continue
|
||||
if i > 7:
|
||||
break
|
||||
result.append(i)
|
||||
print(f"result = {result}")
|
||||
""",
|
||||
)
|
||||
|
||||
# ===== FUNCTIONS =====
|
||||
print("\n📂 FUNCTIONS")
|
||||
|
||||
self.test_feature(
|
||||
"Function Definition",
|
||||
"""
|
||||
def greet(name):
|
||||
return f"Hello, {name}!"
|
||||
|
||||
message = greet("World")
|
||||
print(message)
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Function with Default Args",
|
||||
"""
|
||||
def power(base, exp=2):
|
||||
return base ** exp
|
||||
|
||||
result1 = power(3)
|
||||
result2 = power(3, 4)
|
||||
print(f"3^2 = {result1}, 3^4 = {result2}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Function with *args, **kwargs",
|
||||
"""
|
||||
def flexible_func(*args, **kwargs):
|
||||
return f"args: {args}, kwargs: {kwargs}"
|
||||
|
||||
result = flexible_func(1, 2, 3, name="test", value=42)
|
||||
print(result)
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Lambda Functions",
|
||||
"""
|
||||
square = lambda x: x ** 2
|
||||
numbers = [1, 2, 3, 4, 5]
|
||||
squared = list(map(square, numbers))
|
||||
print(f"squared = {squared}")
|
||||
""",
|
||||
)
|
||||
|
||||
# ===== CLASSES =====
|
||||
print("\n📂 CLASSES")
|
||||
|
||||
self.test_feature(
|
||||
"Class Definition",
|
||||
"""
|
||||
class Counter:
|
||||
def __init__(self, start=0):
|
||||
self.value = start
|
||||
|
||||
def increment(self):
|
||||
self.value += 1
|
||||
return self.value
|
||||
|
||||
counter = Counter(10)
|
||||
result = counter.increment()
|
||||
print(f"counter value = {result}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Class Inheritance",
|
||||
"""
|
||||
class Animal:
|
||||
def speak(self):
|
||||
return "Some sound"
|
||||
|
||||
class Dog(Animal):
|
||||
def speak(self):
|
||||
return "Woof!"
|
||||
|
||||
dog = Dog()
|
||||
print(dog.speak())
|
||||
""",
|
||||
)
|
||||
|
||||
# ===== EXCEPTION HANDLING =====
|
||||
print("\n📂 EXCEPTION HANDLING")
|
||||
|
||||
self.test_feature(
|
||||
"Try/Except",
|
||||
"""
|
||||
try:
|
||||
result = 10 / 2
|
||||
print(f"Division result: {result}")
|
||||
except ZeroDivisionError:
|
||||
print("Cannot divide by zero!")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Try/Except/Finally",
|
||||
"""
|
||||
try:
|
||||
x = int("42")
|
||||
print(f"Parsed: {x}")
|
||||
except ValueError:
|
||||
print("Invalid number")
|
||||
finally:
|
||||
print("Cleanup completed")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Raise Exception",
|
||||
"""
|
||||
try:
|
||||
raise ValueError("Custom error message")
|
||||
except ValueError as e:
|
||||
print(f"Caught: {e}")
|
||||
""",
|
||||
)
|
||||
|
||||
# ===== ADVANCED FEATURES =====
|
||||
print("\n📂 ADVANCED FEATURES")
|
||||
|
||||
self.test_feature(
|
||||
"List Comprehension",
|
||||
"""
|
||||
numbers = [1, 2, 3, 4, 5]
|
||||
squares = [x**2 for x in numbers if x % 2 == 1]
|
||||
print(f"odd squares = {squares}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Dictionary Comprehension",
|
||||
"""
|
||||
numbers = [1, 2, 3, 4, 5]
|
||||
square_dict = {x: x**2 for x in numbers}
|
||||
print(f"square_dict = {square_dict}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Generator Expression",
|
||||
"""
|
||||
numbers = [1, 2, 3, 4, 5]
|
||||
squares_gen = (x**2 for x in numbers)
|
||||
squares_list = list(squares_gen)
|
||||
print(f"squares = {squares_list}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"With Statement",
|
||||
"""
|
||||
class TestContext:
|
||||
def __enter__(self):
|
||||
print("Entering context")
|
||||
return self
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
print("Exiting context")
|
||||
|
||||
with TestContext() as ctx:
|
||||
print("Inside context")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Yield (Generator)",
|
||||
"""
|
||||
def count_up_to(max_count):
|
||||
count = 1
|
||||
while count <= max_count:
|
||||
yield count
|
||||
count += 1
|
||||
|
||||
result = list(count_up_to(3))
|
||||
print(f"generated = {result}")
|
||||
""",
|
||||
)
|
||||
|
||||
# ===== IMPORTS =====
|
||||
print("\n📂 IMPORTS")
|
||||
|
||||
self.test_feature(
|
||||
"Import Statement",
|
||||
"""
|
||||
import math
|
||||
result = math.sqrt(16)
|
||||
print(f"sqrt(16) = {result}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"From Import",
|
||||
"""
|
||||
from math import pi, cos
|
||||
result = cos(pi)
|
||||
print(f"cos(pi) = {result}")
|
||||
""",
|
||||
)
|
||||
|
||||
# ===== ASYNC/AWAIT =====
|
||||
print("\n📂 ASYNC/AWAIT")
|
||||
|
||||
self.test_feature(
|
||||
"Async Function Definition",
|
||||
"""
|
||||
async def async_greet(name):
|
||||
return f"Hello, {name}!"
|
||||
|
||||
# Note: This may not work without event loop
|
||||
print("Async function defined")
|
||||
""",
|
||||
"May fail without event loop",
|
||||
)
|
||||
|
||||
# ===== MATCH STATEMENTS (Python 3.10+) =====
|
||||
print("\n📂 PATTERN MATCHING (Python 3.10+)")
|
||||
|
||||
self.test_feature(
|
||||
"Match Statement",
|
||||
"""
|
||||
def describe_animal(animal):
|
||||
match animal:
|
||||
case "dog":
|
||||
return "loyal companion"
|
||||
case "cat":
|
||||
return "independent hunter"
|
||||
case _:
|
||||
return "unknown animal"
|
||||
|
||||
result = describe_animal("dog")
|
||||
print(f"dog is a {result}")
|
||||
""",
|
||||
"May fail in older Python versions",
|
||||
)
|
||||
|
||||
# ===== TYPE ANNOTATIONS =====
|
||||
print("\n📂 TYPE ANNOTATIONS")
|
||||
|
||||
self.test_feature(
|
||||
"Function Type Annotations",
|
||||
"""
|
||||
def add_numbers(a: int, b: int) -> int:
|
||||
return a + b
|
||||
|
||||
result = add_numbers(5, 3)
|
||||
print(f"5 + 3 = {result}")
|
||||
""",
|
||||
)
|
||||
|
||||
self.test_feature(
|
||||
"Variable Type Annotations",
|
||||
"""
|
||||
name: str = "Alice"
|
||||
age: int = 30
|
||||
height: float = 5.6
|
||||
print(f"{name} is {age} years old and {height} feet tall")
|
||||
""",
|
||||
)
|
||||
|
||||
# ===== OPERATOR OVERLOADING =====
|
||||
print("\n📂 OPERATOR OVERLOADING")
|
||||
|
||||
self.test_feature(
|
||||
"Custom Operators",
|
||||
"""
|
||||
class Vector:
|
||||
def __init__(self, x, y):
|
||||
self.x, self.y = x, y
|
||||
|
||||
def __add__(self, other):
|
||||
return Vector(self.x + other.x, self.y + other.y)
|
||||
|
||||
def __str__(self):
|
||||
return f"Vector({self.x}, {self.y})"
|
||||
|
||||
v1 = Vector(1, 2)
|
||||
v2 = Vector(3, 4)
|
||||
v3 = v1 + v2
|
||||
print(f"v1 + v2 = {v3}")
|
||||
""",
|
||||
)
|
||||
|
||||
def print_summary(self):
|
||||
"""Print audit results summary"""
|
||||
print("\n" + "=" * 60)
|
||||
print("📊 AUDIT SUMMARY")
|
||||
print("=" * 60)
|
||||
|
||||
passed = sum(1 for r in self.results.values() if r["status"] == "PASSED")
|
||||
failed = sum(1 for r in self.results.values() if r["status"] == "FAILED")
|
||||
exceptions = sum(1 for r in self.results.values() if r["status"] == "EXCEPTION")
|
||||
total = len(self.results)
|
||||
|
||||
print(f"Total Features Tested: {total}")
|
||||
print(f"✅ Passed: {passed}")
|
||||
print(f"❌ Failed: {failed}")
|
||||
print(f"💥 Exceptions: {exceptions}")
|
||||
print(f"Success Rate: {passed / total * 100:.1f}%")
|
||||
|
||||
if failed > 0 or exceptions > 0:
|
||||
print("\n🚨 ISSUES FOUND:")
|
||||
for name, result in self.results.items():
|
||||
if result["status"] != "PASSED":
|
||||
status_icon = "❌" if result["status"] == "FAILED" else "💥"
|
||||
print(f" {status_icon} {name}")
|
||||
if isinstance(result["error"], str):
|
||||
print(f" Error: {result['error'][:100]}...")
|
||||
|
||||
def cleanup(self):
|
||||
"""Clean up resources"""
|
||||
if self.instance:
|
||||
self.instance.cleanup()
|
||||
|
||||
|
||||
def main():
|
||||
"""Main function to run the AST audit"""
|
||||
auditor = ASTFeatureAuditor()
|
||||
|
||||
if not auditor.setup_instance():
|
||||
print("Cannot proceed without Factorio instance")
|
||||
return
|
||||
|
||||
try:
|
||||
auditor.run_comprehensive_audit()
|
||||
auditor.print_summary()
|
||||
finally:
|
||||
auditor.cleanup()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,319 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Detailed analysis of missing AST node types in FLE namespace.
|
||||
|
||||
This script identifies specific AST node types that are not explicitly handled
|
||||
in the FLE execute_node method.
|
||||
"""
|
||||
|
||||
import ast
|
||||
import inspect
|
||||
|
||||
|
||||
def analyze_missing_ast_features():
|
||||
"""Analyze which AST node types are missing from FLE implementation"""
|
||||
|
||||
print("🔍 AST NODE TYPE ANALYSIS")
|
||||
print("=" * 60)
|
||||
|
||||
# Get all AST node types from the ast module
|
||||
all_ast_nodes = []
|
||||
for name in dir(ast):
|
||||
obj = getattr(ast, name)
|
||||
if inspect.isclass(obj) and issubclass(obj, ast.AST):
|
||||
all_ast_nodes.append(name)
|
||||
|
||||
print(f"Total AST node types in Python: {len(all_ast_nodes)}")
|
||||
|
||||
# Categorize AST nodes
|
||||
statements = []
|
||||
expressions = []
|
||||
others = []
|
||||
|
||||
for node_name in all_ast_nodes:
|
||||
node_class = getattr(ast, node_name)
|
||||
if hasattr(ast, "stmt") and issubclass(node_class, ast.stmt):
|
||||
statements.append(node_name)
|
||||
elif hasattr(ast, "expr") and issubclass(node_class, ast.expr):
|
||||
expressions.append(node_name)
|
||||
else:
|
||||
others.append(node_name)
|
||||
|
||||
# Currently implemented in FLE (from our analysis)
|
||||
implemented_in_fle = [
|
||||
"Break",
|
||||
"Continue",
|
||||
"For",
|
||||
"While",
|
||||
"If",
|
||||
"FunctionDef",
|
||||
"Assign",
|
||||
"AnnAssign",
|
||||
"AugAssign",
|
||||
"Expr",
|
||||
"Try",
|
||||
]
|
||||
|
||||
print("\n📋 STATEMENT NODES:")
|
||||
print(f"Total: {len(statements)}")
|
||||
for stmt in sorted(statements):
|
||||
status = "✅" if stmt in implemented_in_fle else "❌"
|
||||
print(f" {status} ast.{stmt}")
|
||||
|
||||
print("\n📋 EXPRESSION NODES:")
|
||||
print(f"Total: {len(expressions)}")
|
||||
for expr in sorted(expressions):
|
||||
status = "🔄" # Expressions are handled generically
|
||||
print(f" {status} ast.{expr}")
|
||||
|
||||
print("\n📋 OTHER NODES:")
|
||||
print(f"Total: {len(others)}")
|
||||
for other in sorted(others):
|
||||
print(f" 📝 ast.{other}")
|
||||
|
||||
# Identify missing statement handlers
|
||||
missing_statements = [stmt for stmt in statements if stmt not in implemented_in_fle]
|
||||
|
||||
print(f"\n🚨 MISSING STATEMENT HANDLERS: {len(missing_statements)}")
|
||||
print("=" * 60)
|
||||
|
||||
if missing_statements:
|
||||
critical_missing = []
|
||||
moderate_missing = []
|
||||
minor_missing = []
|
||||
|
||||
for stmt in missing_statements:
|
||||
if stmt in [
|
||||
"Return",
|
||||
"Raise",
|
||||
"Assert",
|
||||
"Import",
|
||||
"ImportFrom",
|
||||
"Global",
|
||||
"Nonlocal",
|
||||
]:
|
||||
critical_missing.append(stmt)
|
||||
elif stmt in [
|
||||
"With",
|
||||
"AsyncWith",
|
||||
"AsyncFor",
|
||||
"AsyncFunctionDef",
|
||||
"ClassDef",
|
||||
]:
|
||||
moderate_missing.append(stmt)
|
||||
else:
|
||||
minor_missing.append(stmt)
|
||||
|
||||
if critical_missing:
|
||||
print(f"\n🔥 CRITICAL MISSING ({len(critical_missing)}):")
|
||||
for stmt in critical_missing:
|
||||
print(f" ❌ ast.{stmt} - {get_statement_description(stmt)}")
|
||||
|
||||
if moderate_missing:
|
||||
print(f"\n⚠️ MODERATE MISSING ({len(moderate_missing)}):")
|
||||
for stmt in moderate_missing:
|
||||
print(f" ❌ ast.{stmt} - {get_statement_description(stmt)}")
|
||||
|
||||
if minor_missing:
|
||||
print(f"\n💭 MINOR MISSING ({len(minor_missing)}):")
|
||||
for stmt in minor_missing:
|
||||
print(f" ❌ ast.{stmt} - {get_statement_description(stmt)}")
|
||||
|
||||
return missing_statements, implemented_in_fle
|
||||
|
||||
|
||||
def get_statement_description(stmt_name):
|
||||
"""Get description of what each AST statement does"""
|
||||
descriptions = {
|
||||
"Return": "Function return statements",
|
||||
"Raise": "Exception raising",
|
||||
"Assert": "Assertion statements",
|
||||
"Import": "Import statements (import module)",
|
||||
"ImportFrom": "From-import statements (from module import name)",
|
||||
"Global": "Global variable declarations",
|
||||
"Nonlocal": "Nonlocal variable declarations",
|
||||
"With": "Context manager statements (with...)",
|
||||
"AsyncWith": "Async context managers (async with...)",
|
||||
"AsyncFor": "Async for loops (async for...)",
|
||||
"AsyncFunctionDef": "Async function definitions (async def...)",
|
||||
"ClassDef": "Class definitions",
|
||||
"Delete": "Delete statements (del...)",
|
||||
"Pass": "Pass statements (no-op)",
|
||||
"ExprStmt": "Expression statements",
|
||||
"Match": "Pattern matching (match/case) - Python 3.10+",
|
||||
}
|
||||
return descriptions.get(stmt_name, "Unknown statement type")
|
||||
|
||||
|
||||
def analyze_lambda_issue():
|
||||
"""Analyze the specific lambda function issue found in audit"""
|
||||
print("\n🔍 LAMBDA FUNCTION ISSUE ANALYSIS")
|
||||
print("=" * 60)
|
||||
|
||||
print("The lambda function test failed with: KeyError: 'args'")
|
||||
print("This suggests an issue in function argument processing in FLE.")
|
||||
print(
|
||||
"\nLambda functions create ast.Lambda nodes, which are expressions, not statements."
|
||||
)
|
||||
print(
|
||||
"The issue is likely in how FLE handles function calls with lambda arguments."
|
||||
)
|
||||
|
||||
# Test code that failed:
|
||||
failed_code = """
|
||||
square = lambda x: x ** 2
|
||||
numbers = [1, 2, 3, 4, 5]
|
||||
squared = list(map(square, numbers))
|
||||
print(f"squared = {squared}")
|
||||
"""
|
||||
|
||||
print("\nFailed code AST analysis:")
|
||||
tree = ast.parse(failed_code)
|
||||
|
||||
for i, node in enumerate(tree.body):
|
||||
print(f" Line {i + 1}: {type(node).__name__}")
|
||||
if isinstance(node, ast.Assign) and isinstance(node.value, ast.Lambda):
|
||||
print(f" Contains ast.Lambda: args={node.value.args}")
|
||||
elif isinstance(node, ast.Assign) and isinstance(node.value, ast.Call):
|
||||
print(f" Contains ast.Call: {ast.unparse(node.value)}")
|
||||
|
||||
|
||||
def provide_implementation_recommendations():
|
||||
"""Provide specific recommendations for implementing missing features"""
|
||||
print("\n💡 IMPLEMENTATION RECOMMENDATIONS")
|
||||
print("=" * 60)
|
||||
|
||||
recommendations = [
|
||||
{
|
||||
"feature": "ast.Return",
|
||||
"priority": "HIGH",
|
||||
"description": "Return statements in functions",
|
||||
"implementation": """
|
||||
elif isinstance(node, ast.Return):
|
||||
if node.value:
|
||||
return_value = eval(compile(ast.Expression(node.value), "file", "eval"), eval_dict)
|
||||
return ('RETURN', return_value)
|
||||
else:
|
||||
return ('RETURN', None)
|
||||
""",
|
||||
},
|
||||
{
|
||||
"feature": "ast.Raise",
|
||||
"priority": "HIGH",
|
||||
"description": "Exception raising",
|
||||
"implementation": """
|
||||
elif isinstance(node, ast.Raise):
|
||||
if node.exc:
|
||||
exception = eval(compile(ast.Expression(node.exc), "file", "eval"), eval_dict)
|
||||
if node.cause:
|
||||
cause = eval(compile(ast.Expression(node.cause), "file", "eval"), eval_dict)
|
||||
raise exception from cause
|
||||
else:
|
||||
raise exception
|
||||
else:
|
||||
raise # Re-raise current exception
|
||||
""",
|
||||
},
|
||||
{
|
||||
"feature": "ast.Assert",
|
||||
"priority": "MEDIUM",
|
||||
"description": "Assertion statements",
|
||||
"implementation": """
|
||||
elif isinstance(node, ast.Assert):
|
||||
test_result = eval(compile(ast.Expression(node.test), "file", "eval"), eval_dict)
|
||||
if not test_result:
|
||||
if node.msg:
|
||||
msg = eval(compile(ast.Expression(node.msg), "file", "eval"), eval_dict)
|
||||
raise AssertionError(msg)
|
||||
else:
|
||||
raise AssertionError()
|
||||
""",
|
||||
},
|
||||
{
|
||||
"feature": "ast.Import",
|
||||
"priority": "MEDIUM",
|
||||
"description": "Import statements",
|
||||
"implementation": """
|
||||
elif isinstance(node, ast.Import):
|
||||
for alias in node.names:
|
||||
module = __import__(alias.name)
|
||||
name = alias.asname if alias.asname else alias.name
|
||||
eval_dict[name] = module
|
||||
self.persistent_vars[name] = module
|
||||
setattr(self, name, module)
|
||||
""",
|
||||
},
|
||||
{
|
||||
"feature": "ast.ImportFrom",
|
||||
"priority": "MEDIUM",
|
||||
"description": "From-import statements",
|
||||
"implementation": """
|
||||
elif isinstance(node, ast.ImportFrom):
|
||||
module = __import__(node.module, fromlist=[alias.name for alias in node.names])
|
||||
for alias in node.names:
|
||||
obj = getattr(module, alias.name)
|
||||
name = alias.asname if alias.asname else alias.name
|
||||
eval_dict[name] = obj
|
||||
self.persistent_vars[name] = obj
|
||||
setattr(self, name, obj)
|
||||
""",
|
||||
},
|
||||
{
|
||||
"feature": "ast.With",
|
||||
"priority": "LOW",
|
||||
"description": "Context manager statements",
|
||||
"implementation": """
|
||||
elif isinstance(node, ast.With):
|
||||
# Context manager implementation is complex
|
||||
# May require significant changes to execution model
|
||||
# Consider falling back to generic exec() for now
|
||||
compiled = compile(ast.Module([node], type_ignores=[]), "file", "exec")
|
||||
exec(compiled, eval_dict)
|
||||
""",
|
||||
},
|
||||
{
|
||||
"feature": "Lambda Functions Bug Fix",
|
||||
"priority": "HIGH",
|
||||
"description": "Fix lambda function argument processing",
|
||||
"implementation": """
|
||||
# The lambda issue is likely in function call handling
|
||||
# Need to fix the argument processing in ast.Expr -> ast.Call handling
|
||||
# Look for 'args' key access that's failing and add proper error handling
|
||||
""",
|
||||
},
|
||||
]
|
||||
|
||||
for rec in recommendations:
|
||||
priority_icon = {"HIGH": "🔥", "MEDIUM": "⚠️", "LOW": "💭"}[rec["priority"]]
|
||||
print(f"\n{priority_icon} {rec['feature']} ({rec['priority']} PRIORITY)")
|
||||
print(f" {rec['description']}")
|
||||
if "implementation" in rec:
|
||||
print(" Implementation snippet:")
|
||||
for line in rec["implementation"].strip().split("\n"):
|
||||
print(f" {line}")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main analysis function"""
|
||||
missing, implemented = analyze_missing_ast_features()
|
||||
analyze_lambda_issue()
|
||||
provide_implementation_recommendations()
|
||||
|
||||
print("\n🎯 SUMMARY")
|
||||
print("=" * 60)
|
||||
print(f"• FLE currently handles {len(implemented)} statement types explicitly")
|
||||
print(f"• {len(missing)} statement types are missing explicit handlers")
|
||||
print("• Most expressions work through generic eval() fallback")
|
||||
print("• 2 specific bugs identified: Lambda functions & false positive on Raise")
|
||||
print("• Overall language support: 93.1% (very good!)")
|
||||
|
||||
print("\n🛠️ RECOMMENDED ACTIONS:")
|
||||
print("1. Fix lambda function argument processing bug (HIGH)")
|
||||
print("2. Add ast.Return, ast.Raise, ast.Assert handlers (HIGH)")
|
||||
print("3. Add ast.Import, ast.ImportFrom handlers (MEDIUM)")
|
||||
print("4. Consider ast.With, ast.ClassDef handlers (LOW)")
|
||||
print("5. The fallback exec() handles most missing cases adequately")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,601 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Debug script to investigate the iron plate extraction discrepancy in v4168, agent0, iter18.
|
||||
|
||||
The program shows extracted plates (40+30+30=100) in print statements, but total_plates ends up as 0.
|
||||
This script recreates the exact scenario from the database to debug what happened.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import sys
|
||||
from typing import Optional
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Add the project root to Python path
|
||||
sys.path.append("/Users/neel/Desktop/Work/factorio-learning-environment")
|
||||
|
||||
from fle.commons.db_client import create_db_client, PostgresDBClient
|
||||
from fle.commons.models.game_state import GameState
|
||||
from fle.env import FactorioInstance
|
||||
from fle.env.game_types import Prototype
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
class IronPlateDebugger:
|
||||
"""Debug class to investigate iron plate extraction issue"""
|
||||
|
||||
def __init__(self):
|
||||
self.db_client = None
|
||||
self.game_instance = None
|
||||
self.version = 4168
|
||||
self.agent_idx = 0
|
||||
self.iteration = 18
|
||||
|
||||
async def setup_db_client(self):
|
||||
"""Initialize database client"""
|
||||
print("Setting up database client...")
|
||||
try:
|
||||
self.db_client = await create_db_client()
|
||||
print(f"✓ Database client initialized: {type(self.db_client).__name__}")
|
||||
except Exception as e:
|
||||
print(f"✗ Failed to initialize database: {e}")
|
||||
raise
|
||||
|
||||
async def get_game_state_from_db(self) -> Optional[GameState]:
|
||||
"""Retrieve the game state for the specific iteration from database"""
|
||||
print(
|
||||
f"Retrieving game state for version={self.version}, agent={self.agent_idx}, iteration={self.iteration}"
|
||||
)
|
||||
|
||||
try:
|
||||
if isinstance(self.db_client, PostgresDBClient):
|
||||
hack_query = """
|
||||
SELECT state_json, code, response, achievements_json FROM programs WHERE id = 577121 LIMIT 1
|
||||
"""
|
||||
query = """
|
||||
SELECT state_json, code, response, achievements_json FROM programs
|
||||
WHERE version = %s AND instance = %s AND depth = %s
|
||||
AND state_json IS NOT NULL
|
||||
LIMIT 1
|
||||
"""
|
||||
with self.db_client.get_connection() as conn:
|
||||
with conn.cursor() as cur:
|
||||
cur.execute(
|
||||
hack_query, (self.version, self.agent_idx, self.iteration)
|
||||
)
|
||||
result = cur.fetchone()
|
||||
else: # SQLite
|
||||
query = """
|
||||
SELECT state_json, code, response, achievements_json FROM programs
|
||||
WHERE version = ? AND instance = ? AND depth = ?
|
||||
AND state_json IS NOT NULL
|
||||
LIMIT 1
|
||||
"""
|
||||
with self.db_client.get_connection() as conn:
|
||||
cur = conn.cursor()
|
||||
cur.execute(query, (self.version, self.agent_idx, self.iteration))
|
||||
result = cur.fetchone()
|
||||
|
||||
if not result:
|
||||
print(
|
||||
f"✗ No program found for version={self.version}, agent={self.agent_idx}, iteration={self.iteration}"
|
||||
)
|
||||
return None
|
||||
|
||||
# Parse the JSON state
|
||||
game_state = GameState.parse(result[0])
|
||||
|
||||
print("✓ Retrieved game state from database")
|
||||
# print(f" - Game tick: {game_state.game_tick}")
|
||||
# print(f" - Score: {game_state.score}")
|
||||
print(f" - Program code length: {len(result[1])} chars")
|
||||
print(f" - Response length: {len(result[2])} chars")
|
||||
|
||||
return game_state
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Error retrieving game state: {e}")
|
||||
raise
|
||||
|
||||
def setup_factorio_instance(self):
|
||||
"""Initialize Factorio instance for debugging"""
|
||||
print("Setting up Factorio instance...")
|
||||
try:
|
||||
# Use localhost setup (assuming docker container is running)
|
||||
self.game_instance = FactorioInstance(
|
||||
address="localhost",
|
||||
tcp_port=27000, # Default port for first container
|
||||
num_agents=1,
|
||||
fast=True,
|
||||
cache_scripts=True,
|
||||
inventory={},
|
||||
all_technologies_researched=True,
|
||||
bounding_box=200,
|
||||
)
|
||||
|
||||
# Speed up the game for debugging
|
||||
self.game_instance.set_speed_and_unpause(10)
|
||||
print("✓ Factorio instance initialized")
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Failed to initialize Factorio instance: {e}")
|
||||
print("Make sure a Factorio docker container is running on port 27000")
|
||||
raise
|
||||
|
||||
def load_game_state(self, game_state: GameState):
|
||||
"""Load the specific game state into the Factorio instance"""
|
||||
print("Loading game state into Factorio instance...")
|
||||
try:
|
||||
self.game_instance.reset(game_state)
|
||||
print("✓ Game state loaded successfully")
|
||||
import time
|
||||
|
||||
time.sleep(0.4)
|
||||
|
||||
# Verify the state was loaded correctly
|
||||
current_score, _ = self.game_instance.namespaces[0].score()
|
||||
print(f" - Current score: {current_score}")
|
||||
# print(f" - Expected score: {game_state.score}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Failed to load game state: {e}")
|
||||
raise
|
||||
|
||||
def debug_program_execution(self, original_game_state):
|
||||
"""Execute the program step by step with debugging"""
|
||||
print("\n" + "=" * 60)
|
||||
print("DEBUGGING PROGRAM EXECUTION")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
# Get the namespace for API calls
|
||||
namespace = self.game_instance.namespaces[0]
|
||||
|
||||
print("\n1. Getting all stone furnaces...")
|
||||
furnaces = namespace.get_entities({Prototype.StoneFurnace})
|
||||
print(f" Found {len(furnaces)} stone furnaces:")
|
||||
for i, furnace in enumerate(furnaces):
|
||||
print(f" - Furnace {i + 1}: {furnace.position}")
|
||||
|
||||
print("\n2. Inspecting furnace inventories...")
|
||||
total_plates_before = 0
|
||||
furnace_inventories = []
|
||||
|
||||
for i, furnace in enumerate(furnaces):
|
||||
inventory = namespace.inspect_inventory(furnace)
|
||||
plates = inventory.get(Prototype.IronPlate, 0)
|
||||
total_plates_before += plates
|
||||
furnace_inventories.append((furnace, plates))
|
||||
print(
|
||||
f" - Furnace {i + 1} at {furnace.position}: {plates} iron plates"
|
||||
)
|
||||
|
||||
print(f"\n Total iron plates in all furnaces: {total_plates_before}")
|
||||
|
||||
print("\n3. Extracting iron plates...")
|
||||
total_plates_extracted = 0
|
||||
extraction_results = []
|
||||
|
||||
for i, (furnace, plates_available) in enumerate(furnace_inventories):
|
||||
if plates_available > 0:
|
||||
print(
|
||||
f"\n Extracting from furnace {i + 1} at {furnace.position}:"
|
||||
)
|
||||
print(f" - Plates available: {plates_available}")
|
||||
|
||||
try:
|
||||
# This is the exact line from the program
|
||||
extracted = namespace.extract_item(
|
||||
Prototype.IronPlate, furnace, quantity=plates_available
|
||||
)
|
||||
print(f" - Plates extracted: {extracted}")
|
||||
total_plates_extracted += extracted
|
||||
extraction_results.append(
|
||||
(furnace, plates_available, extracted)
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
print(f" - ✗ Extraction failed: {e}")
|
||||
extraction_results.append((furnace, plates_available, 0))
|
||||
else:
|
||||
print(f" - Furnace {i + 1}: No plates to extract")
|
||||
|
||||
print("\n4. RESULTS SUMMARY:")
|
||||
print(f" - Total plates before extraction: {total_plates_before}")
|
||||
print(f" - Total plates extracted: {total_plates_extracted}")
|
||||
print(f" - Discrepancy: {total_plates_before - total_plates_extracted}")
|
||||
|
||||
print("\n5. DETAILED EXTRACTION RESULTS:")
|
||||
for i, (furnace, available, extracted) in enumerate(extraction_results):
|
||||
print(
|
||||
f" - Furnace {i + 1} at {furnace.position}: {available} available → {extracted} extracted"
|
||||
)
|
||||
|
||||
# Check player inventory
|
||||
print("\n6. PLAYER INVENTORY CHECK:")
|
||||
player_inventory = namespace.inspect_inventory()
|
||||
player_iron_plates = player_inventory.get(Prototype.IronPlate, 0)
|
||||
print(f" - Iron plates in player inventory: {player_iron_plates}")
|
||||
|
||||
# Re-check furnace inventories after extraction
|
||||
print("\n7. POST-EXTRACTION FURNACE INVENTORIES:")
|
||||
total_plates_after = 0
|
||||
for i, furnace in enumerate(furnaces):
|
||||
inventory = namespace.inspect_inventory(furnace)
|
||||
plates = inventory.get(Prototype.IronPlate, 0)
|
||||
total_plates_after += plates
|
||||
print(
|
||||
f" - Furnace {i + 1} at {furnace.position}: {plates} iron plates remaining"
|
||||
)
|
||||
|
||||
print(f" - Total plates remaining in furnaces: {total_plates_after}")
|
||||
|
||||
# Simulate the original program's total_plates variable
|
||||
print("\n8. SIMULATING ORIGINAL PROGRAM LOGIC:")
|
||||
print(" Original code:")
|
||||
print(" total_plates = 0")
|
||||
print(" for furnace in furnaces:")
|
||||
print(" plates = inspect_inventory(furnace)[Prototype.IronPlate]")
|
||||
print(" if plates > 0:")
|
||||
print(
|
||||
" extracted = extract_item(Prototype.IronPlate, furnace, quantity=plates)"
|
||||
)
|
||||
print(" total_plates += extracted")
|
||||
print(
|
||||
" print(f'Extracted {extracted} iron plates from furnace at {furnace.position}')"
|
||||
)
|
||||
print("")
|
||||
print(" Simulation:")
|
||||
|
||||
simulated_total = 0
|
||||
for i, (furnace, available, extracted) in enumerate(extraction_results):
|
||||
if available > 0:
|
||||
simulated_total += extracted
|
||||
print(
|
||||
f" total_plates += {extracted} # (now total_plates = {simulated_total})"
|
||||
)
|
||||
|
||||
print(f"\n Final simulated total_plates: {simulated_total}")
|
||||
|
||||
# Test the exact accumulation logic that's failing
|
||||
print("\n9. TESTING EXACT ACCUMULATION LOGIC:")
|
||||
print(" Testing the exact += operation that seems to be failing...")
|
||||
|
||||
test_total_plates = 0
|
||||
print(f" Initial total_plates: {test_total_plates}")
|
||||
|
||||
for i, (furnace, plates_available, extracted) in enumerate(
|
||||
extraction_results
|
||||
):
|
||||
if plates_available > 0:
|
||||
print(
|
||||
f" Before += operation: total_plates = {test_total_plates}, extracted = {extracted}"
|
||||
)
|
||||
print(f" Executing: total_plates += {extracted}")
|
||||
|
||||
# Store previous value to detect any issues
|
||||
prev_total = test_total_plates
|
||||
test_total_plates += extracted
|
||||
|
||||
print(f" After += operation: total_plates = {test_total_plates}")
|
||||
|
||||
# Verify the operation worked correctly
|
||||
expected = prev_total + extracted
|
||||
if test_total_plates != expected:
|
||||
print(
|
||||
f" ⚠️ ACCUMULATION FAILURE: Expected {expected}, got {test_total_plates}"
|
||||
)
|
||||
else:
|
||||
print(" ✓ Accumulation successful")
|
||||
print()
|
||||
|
||||
print(f" Final test total_plates: {test_total_plates}")
|
||||
|
||||
# Reset game state before testing FLE execution environment
|
||||
print("\n9.5. RESETTING GAME STATE FOR FLE TESTING:")
|
||||
print(" Reloading original game state to restore iron plates...")
|
||||
try:
|
||||
# Use the original game state that was passed in
|
||||
if original_game_state:
|
||||
self.load_game_state(original_game_state)
|
||||
print(" ✓ Game state reset successfully")
|
||||
|
||||
# Verify the reset worked
|
||||
namespace = self.game_instance.namespaces[0]
|
||||
furnaces = namespace.get_entities({Prototype.StoneFurnace})
|
||||
total_plates_after_reset = 0
|
||||
for i, furnace in enumerate(furnaces):
|
||||
inventory = namespace.inspect_inventory(furnace)
|
||||
plates = inventory.get(Prototype.IronPlate, 0)
|
||||
total_plates_after_reset += plates
|
||||
print(
|
||||
f" - Furnace {i + 1} at {furnace.position}: {plates} iron plates"
|
||||
)
|
||||
print(f" - Total plates after reset: {total_plates_after_reset}")
|
||||
else:
|
||||
print(" ✗ No original game state available")
|
||||
except Exception as e:
|
||||
print(f" ✗ Failed to reset game state: {e}")
|
||||
|
||||
# Test using the actual FLE execution environment
|
||||
print("\n10. TESTING IN ACTUAL FLE EXECUTION ENVIRONMENT:")
|
||||
print(
|
||||
" Running the exact original program code using instance.eval_with_error()..."
|
||||
)
|
||||
|
||||
# The exact original program code
|
||||
original_program = """
|
||||
# Extract iron plates from all furnaces
|
||||
total_plates = 0
|
||||
furnaces = get_entities({Prototype.StoneFurnace})
|
||||
for furnace in furnaces:
|
||||
plates = inspect_inventory(furnace)[Prototype.IronPlate]
|
||||
if plates > 0:
|
||||
extracted = extract_item(Prototype.IronPlate, furnace, quantity=plates)
|
||||
total_plates += extracted
|
||||
print(f"Extracted {extracted} iron plates from furnace at {furnace.position}")
|
||||
|
||||
print(f"Total iron plates extracted: {total_plates}")
|
||||
total_plates # Return the final value
|
||||
"""
|
||||
|
||||
print(" Executing original program in FLE environment...")
|
||||
try:
|
||||
result = self.game_instance.eval_with_error(
|
||||
original_program, agent_idx=0, timeout=60
|
||||
)
|
||||
print(f" FLE execution result: {result}")
|
||||
|
||||
# result should be (return_code, stdout, stderr)
|
||||
if len(result) >= 3:
|
||||
return_code, stdout, stderr = result[0], result[1], result[2]
|
||||
print(f" Return code: {return_code}")
|
||||
print(f" Stdout: {stdout}")
|
||||
print(f" Stderr: {stderr}")
|
||||
|
||||
# The final value of total_plates should be in the result
|
||||
print(f" Final total_plates from FLE execution: {result}")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ✗ FLE execution failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
|
||||
# Test step-by-step execution in FLE - FOCUSED ON += BUG
|
||||
print("\n11. FOCUSED += OPERATION TEST IN FLE:")
|
||||
print(" Testing if += operations work correctly in FLE...")
|
||||
|
||||
try:
|
||||
# Test simple accumulation first
|
||||
print("\n A. Simple accumulation test:")
|
||||
simple_test = self.game_instance.eval_with_error(
|
||||
"""
|
||||
total = 0
|
||||
print(f"Initial total: {total}")
|
||||
total += 40
|
||||
print(f"After adding 40: {total}")
|
||||
total += 30
|
||||
print(f"After adding 30: {total}")
|
||||
total += 30
|
||||
print(f"After adding 30 again: {total}")
|
||||
print(f"Final total: {total}")
|
||||
total
|
||||
""",
|
||||
agent_idx=0,
|
||||
)
|
||||
print(f" Simple accumulation result: {simple_test}")
|
||||
|
||||
# Test with function calls
|
||||
print("\n B. Accumulation with extract_item calls:")
|
||||
extract_test = self.game_instance.eval_with_error(
|
||||
"""
|
||||
total_plates = 0
|
||||
furnaces = get_entities({Prototype.StoneFurnace})
|
||||
print(f"Found {len(furnaces)} furnaces")
|
||||
|
||||
for i, furnace in enumerate(furnaces):
|
||||
plates_available = inspect_inventory(furnace)[Prototype.IronPlate]
|
||||
print(f"Furnace {i+1} has {plates_available} plates")
|
||||
|
||||
if plates_available > 0:
|
||||
print(f"Before extraction - total_plates: {total_plates}")
|
||||
extracted = extract_item(Prototype.IronPlate, furnace, quantity=plates_available)
|
||||
print(f"Extracted: {extracted}")
|
||||
|
||||
print(f"Before += operation - total_plates: {total_plates}, extracted: {extracted}")
|
||||
total_plates += extracted
|
||||
print(f"After += operation - total_plates: {total_plates}")
|
||||
|
||||
print(f"End of iteration {i+1} - total_plates: {total_plates}")
|
||||
|
||||
print(f"FINAL RESULT - total_plates: {total_plates}")
|
||||
total_plates
|
||||
""",
|
||||
agent_idx=0,
|
||||
)
|
||||
print(f" Extract accumulation result: {extract_test}")
|
||||
|
||||
# Test variable persistence across calls
|
||||
print("\n C. Testing variable persistence:")
|
||||
persist1 = self.game_instance.eval_with_error(
|
||||
"test_var = 42", agent_idx=0
|
||||
)
|
||||
print(f" Set test_var = 42: {persist1}")
|
||||
|
||||
persist2 = self.game_instance.eval_with_error("test_var", agent_idx=0)
|
||||
print(f" Check test_var: {persist2}")
|
||||
|
||||
persist3 = self.game_instance.eval_with_error(
|
||||
"test_var += 8", agent_idx=0
|
||||
)
|
||||
print(f" test_var += 8: {persist3}")
|
||||
|
||||
persist4 = self.game_instance.eval_with_error("test_var", agent_idx=0)
|
||||
print(f" Check test_var again: {persist4}")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ✗ Focused += test failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
|
||||
# Test the fix!
|
||||
print("\n12. TESTING THE FIX - POST-PATCH VERIFICATION:")
|
||||
print(" Re-running the original program to see if += now works...")
|
||||
|
||||
try:
|
||||
# Reset game state one more time
|
||||
if original_game_state:
|
||||
self.load_game_state(original_game_state)
|
||||
print(" ✓ Game state reset for fix verification")
|
||||
|
||||
# Run the original program again
|
||||
fixed_result = self.game_instance.eval_with_error(
|
||||
original_program, agent_idx=0, timeout=60
|
||||
)
|
||||
print(f" Fixed program result: {fixed_result}")
|
||||
|
||||
# Parse the output to see if total_plates is now correct
|
||||
stderr_output = fixed_result[2] if len(fixed_result) > 2 else ""
|
||||
|
||||
print("\n Analyzing fixed output:")
|
||||
if "Total iron plates extracted: 0" in stderr_output:
|
||||
print(" ❌ BUG STILL EXISTS - total_plates is still 0")
|
||||
elif "Total iron plates extracted:" in stderr_output:
|
||||
# Extract the actual number
|
||||
lines = stderr_output.split("\n")
|
||||
for line in lines:
|
||||
if "Total iron plates extracted:" in line:
|
||||
print(f" ✅ SUCCESS! Found: {line}")
|
||||
break
|
||||
else:
|
||||
print(" ⚠️ Could not find total_plates output")
|
||||
|
||||
print(" Full stderr output:")
|
||||
for line in stderr_output.split("\n"):
|
||||
if line.strip():
|
||||
print(f" {line}")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ✗ Fix verification failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
|
||||
return {
|
||||
"total_plates_before": total_plates_before,
|
||||
"total_plates_extracted": total_plates_extracted,
|
||||
"player_iron_plates": player_iron_plates,
|
||||
"total_plates_after": total_plates_after,
|
||||
"simulated_total": simulated_total,
|
||||
"extraction_results": extraction_results,
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Error during program execution: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
raise
|
||||
|
||||
async def run_analysis(self):
|
||||
"""Main analysis function"""
|
||||
print("=" * 60)
|
||||
print("IRON PLATE EXTRACTION DEBUG ANALYSIS")
|
||||
print("=" * 60)
|
||||
print(
|
||||
f"Target: Version {self.version}, Agent {self.agent_idx}, Iteration {self.iteration}"
|
||||
)
|
||||
print("")
|
||||
|
||||
try:
|
||||
# Setup
|
||||
await self.setup_db_client()
|
||||
game_state = await self.get_game_state_from_db()
|
||||
|
||||
if not game_state:
|
||||
print("Cannot proceed without game state")
|
||||
return
|
||||
|
||||
self.setup_factorio_instance()
|
||||
self.load_game_state(game_state)
|
||||
|
||||
# Debug the program execution
|
||||
results = self.debug_program_execution(game_state)
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("ANALYSIS CONCLUSION")
|
||||
print("=" * 60)
|
||||
|
||||
# Compare with the original output
|
||||
print("Original program output from observation file:")
|
||||
print("- 'Extracted 40 iron plates from furnace at x=16.0 y=74.0'")
|
||||
print("- 'Extracted 30 iron plates from furnace at x=19.0 y=74.0'")
|
||||
print("- 'Extracted 30 iron plates from furnace at x=22.0 y=74.0'")
|
||||
print("- 'Total iron plates extracted: 0'")
|
||||
print("")
|
||||
|
||||
print("Our debugging results:")
|
||||
print(f"- Total plates extracted: {results['total_plates_extracted']}")
|
||||
print(f"- Simulated total_plates variable: {results['simulated_total']}")
|
||||
print(f"- Player inventory iron plates: {results['player_iron_plates']}")
|
||||
|
||||
if (
|
||||
results["simulated_total"] == 0
|
||||
and results["total_plates_extracted"] > 0
|
||||
):
|
||||
print("\n🔍 LIKELY ISSUE IDENTIFIED:")
|
||||
print(
|
||||
"The extraction operations are succeeding individually, but there may be:"
|
||||
)
|
||||
print("1. A variable scoping issue in the program execution")
|
||||
print(
|
||||
"2. An exception occurring after extraction but before the total is printed"
|
||||
)
|
||||
print("3. The total_plates variable being reset somewhere")
|
||||
print("4. An issue with the program execution environment")
|
||||
elif (
|
||||
results["simulated_total"] == 0
|
||||
and results["total_plates_extracted"] == 0
|
||||
):
|
||||
print("\n🔍 ISSUE IDENTIFIED:")
|
||||
print(
|
||||
"The extract_item calls are failing, despite the print statements suggesting success"
|
||||
)
|
||||
print("This indicates a problem with:")
|
||||
print("1. The extract_item function implementation")
|
||||
print("2. The game state not matching what was expected")
|
||||
print("3. The furnace positions or states being different")
|
||||
|
||||
except Exception as e:
|
||||
print(f"✗ Analysis failed: {e}")
|
||||
import traceback
|
||||
|
||||
traceback.print_exc()
|
||||
|
||||
finally:
|
||||
# Cleanup
|
||||
if self.game_instance:
|
||||
self.game_instance.cleanup()
|
||||
if self.db_client:
|
||||
await self.db_client.cleanup()
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main function to run the debug analysis"""
|
||||
debugger = IronPlateDebugger()
|
||||
await debugger.run_analysis()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("Starting iron plate extraction debug analysis...")
|
||||
print("Make sure you have:")
|
||||
print("1. A Factorio docker container running on port 27000")
|
||||
print("2. Database credentials set in environment variables")
|
||||
print("3. Virtual environment activated")
|
||||
print("")
|
||||
|
||||
asyncio.run(main())
|
||||
Reference in New Issue
Block a user