The same code yields different parsing results.

Lezer
1

I am writing unit tests for my grammar file.

import { parser } from ‘../src-gen/LanguageParser’;

describe(‘Grammar Tests’, () => {
it(‘Single test’, () => {
const code = 
`&ACCESS RVP 
&REL 427 
&COMMENT DAUERLAUF TESTANLAGEN 
&USER ENRICO NICKLISCH 
DEF FileAttributes() 
;Comment 
END;
`
const p = parser.parse(code);
console.log(p.toString());
});
it.each([
‘basic/fileAttributes.txt’,
])(‘%s’, (fileName: string) => {
for (const { run } of generateTest(fileName)) {
run(parser);
}
});
});

function generateTest(fileName: string): fileTestType
 {
const tt = readFileSync(join(krlSourceFileFolder, fileName), ‘utf8’);
return fileTests(tt, fileName);
}

The content of fileAttributes.txt is as follows:

# Test basic file attributes

&ACCESS RVP
&REL 427
&COMMENT DAUERLAUF TESTANLAGEN
&USER ENRICO NICKLISCH
DEF FileAttributes()
;Comment
END

==>

Program(FileAttributeDeclaration(FileAttributeStringParamDecl(FileAttributeAccess,FileAttributeContent)),FileAttributeDeclaration(FileAttributeRelDecl(FileAttributeRel,Number)),FileAttributeDeclaration(FileAttributeStringParamDecl(FileAttributeComment,FileAttributeContent)),FileAttributeDeclaration(FileAttributeStringParamDecl(FileAttributeUser,FileAttributeContent)),ProgramDefinition(ProgramHeader(DEF,Name,BracketLeft,BracketRight),ProgramBody(LineComment,END)))

The code content is exactly the same as in the single test. However, when running the unit test via run(parser), I get an error (I have enabled LOG=parse):

The single unit test result

image
image1913×1081 293 KB

The it.each unit test

image
image1047×227 8.23 KB

Why is this happening?

2

Are you configuring the parser precisely the same in both situations? If you have GLR parsing (~ markers) in your grammar, and multiple parses match the input, it will be hard to predict which one you get, but you should still consistently get the same one when parsing the precise same input. There no nondeterministic code in the parser.

If you can isolate this to something that’s easy to reproduce, I could take a look.

1 Like
3

Hi @marijn , I have prepared a small demo to demonstrate this. The link of the demo is: GitHub - tolerious/lezer-demo · GitHub

And we could run pnpm test after pnpm i, then we can see the issue above.

Thank you for yout time, if you could take a short look at it. :face_with_hand_over_mouth:

4

I’m not sure what I should take from that repo. You’re running two tests with different inputs. Of course those will produce different trees.

5

Yep, I made a mistake before, now it’s the same code. But, it’s interesting, I created a new development environment, and the unit test passed. Let me double check. Thank you so much.

6

Hi @marijn , it’s very interesting, I have updated my code as you said, when I’m running the test by pnpm test it failed on the windows based system, and could succeed on the Linux based system.

7

Found the root cause now.

It’s because of the line break.

After I did like this:

image

Everything is fine. I think this is why the unit test could pass on Linux based system, and fail on windows based system.