Racket

Racket - 使用token-char定义一个字符(Racket - define one character with token-char)

我正在研究一个类的项目，我们的任务是在Python和Racket中编写数字，符号，注释，算术运算符，括号和EOF的扫描程序。 我正在研究球拍版本，我编写了以下一行来定义一个或多个角色作为符号：

[(any-char) (token-CHAR (string->character lexeme))]

我有以下行来定义一个或多个数字作为数字：

[(:+ digit) (token-NUM (string->number lexeme))]

我是Racket的新手，这是我的第三个程序，所以我不确定如何处理这个，所以任何建议都非常感谢。 我已经搜索了Racket文档，但我无法找到我想要的内容。

谢谢！

I am working on a project for a class and we are tasked with writing a scanner for numbers, symbols, comments, arithmetic operators, parenthesis, and EOF in both Python and Racket. I am working on the racket version and I have written the following line to define one or more character as a symbol:

[(any-char) (token-CHAR (string->character lexeme))]

I have the following line to define on or more digits as a number:

[(:+ digit) (token-NUM (string->number lexeme))]

I am very new to Racket, this is my third program, so I am not exactly sure how to approach this, so any suggestions are greatly appreciated. I have scoured the Racket documentation, but I wasn't able to find what I was looking for.

Thanks!

最满意答案

这是一个最小的入门示例 - 评论很多。

#lang racket ;;; IMPORT ;; Import the lexer tools (require parser-tools/yacc parser-tools/lex (prefix-in : parser-tools/lex-sre) ; names from lex-sre are prefixed with : ; to avoid name collisions syntax/readerr) ;;; REGULAR EXPRESSIONS ;; Names for regular expressions matching letters and digits. ;; Note that :or are prefixed with a : due to (prefix-in : ...) above (define-lex-abbrevs [letter (:or (:/ "a" "z") (:/ #\A #\Z) )] [digit (:/ #\0 #\9)]) ;;; TOKENS ;; Tokens such as numbers (and identifiers and strings) carry a value ;; In the example only the NUMBER token is used, but you may need more. (define-tokens value-tokens (NUMBER IDENTIFIER STRING)) ;; Tokens that don't carry a value. (define-empty-tokens op-tokens (newline := = < > + - * / ^ EOF)) ;;; LEXER ;; Here the lexer (aka the scanner) is defined. ;; The construct lexer-src-pos evaluates to a function which scans an input port ;; returning one position-token at a time. ;; A position token contains besides the actual token also source location information ;; (i.e. you can see where in the file the token was read) (define lex (lexer-src-pos [(eof) ; input: eof of file 'EOF] ; output: the symbol EOF [(:or #\tab #\space #\newline) ; input: whitespace (return-without-pos (lex input-port))] ; output: the next token ; (i.e. skip the whitespace) [#\newline ; input: newline (token-newline)] ; ouput: a newline-token ; ; note: (token-newline) returns 'newline [(:or ":=" "+" "-" "*" "/" "^" "<" ">" "=") ; input: an operator (string->symbol lexeme)] ; output: corresponding symbol [(:+ digit) ; input: digits (token-NUMBER (string->number lexeme))])) ; outout: a NUMBER token whose value is ; ; the number ; ; note: (token-value token) ; returns the number ;;; TEST (define input (open-input-string "123+456")) (lex input) ; (position-token (token 'NUMBER 123) (position 1 #f #f) (position 4 #f #f)) (lex input) ; (position-token '+ (position 4 #f #f) (position 5 #f #f)) (lex input) ; (position-token (token 'NUMBER 456) (position 5 #f #f) (position 8 #f #f)) (lex input) ; (position-token 'EOF (position 8 #f #f) (position 8 #f #f)) ;; Let's make it a little easier to play with the lexer. (define (string->tokens s) (port->tokens (open-input-string s))) (define (port->tokens in) (define token (lex in)) (if (eq? (position-token-token token) 'EOF) '() (cons token (port->tokens in)))) (map position-token-token (string->tokens "123*45/3")) ; strip positions ; Output: ; (list (token 'NUMBER 123) ; '* ; (token 'NUMBER 45) ; '/ ; (token 'NUMBER 3))

Here is a minimal getting-started example - heavily commented.

#lang racket ;;; IMPORT ;; Import the lexer tools (require parser-tools/yacc parser-tools/lex (prefix-in : parser-tools/lex-sre) ; names from lex-sre are prefixed with : ; to avoid name collisions syntax/readerr) ;;; REGULAR EXPRESSIONS ;; Names for regular expressions matching letters and digits. ;; Note that :or are prefixed with a : due to (prefix-in : ...) above (define-lex-abbrevs [letter (:or (:/ "a" "z") (:/ #\A #\Z) )] [digit (:/ #\0 #\9)]) ;;; TOKENS ;; Tokens such as numbers (and identifiers and strings) carry a value ;; In the example only the NUMBER token is used, but you may need more. (define-tokens value-tokens (NUMBER IDENTIFIER STRING)) ;; Tokens that don't carry a value. (define-empty-tokens op-tokens (newline := = < > + - * / ^ EOF)) ;;; LEXER ;; Here the lexer (aka the scanner) is defined. ;; The construct lexer-src-pos evaluates to a function which scans an input port ;; returning one position-token at a time. ;; A position token contains besides the actual token also source location information ;; (i.e. you can see where in the file the token was read) (define lex (lexer-src-pos [(eof) ; input: eof of file 'EOF] ; output: the symbol EOF [(:or #\tab #\space #\newline) ; input: whitespace (return-without-pos (lex input-port))] ; output: the next token ; (i.e. skip the whitespace) [#\newline ; input: newline (token-newline)] ; ouput: a newline-token ; ; note: (token-newline) returns 'newline [(:or ":=" "+" "-" "*" "/" "^" "<" ">" "=") ; input: an operator (string->symbol lexeme)] ; output: corresponding symbol [(:+ digit) ; input: digits (token-NUMBER (string->number lexeme))])) ; outout: a NUMBER token whose value is ; ; the number ; ; note: (token-value token) ; returns the number ;;; TEST (define input (open-input-string "123+456")) (lex input) ; (position-token (token 'NUMBER 123) (position 1 #f #f) (position 4 #f #f)) (lex input) ; (position-token '+ (position 4 #f #f) (position 5 #f #f)) (lex input) ; (position-token (token 'NUMBER 456) (position 5 #f #f) (position 8 #f #f)) (lex input) ; (position-token 'EOF (position 8 #f #f) (position 8 #f #f)) ;; Let's make it a little easier to play with the lexer. (define (string->tokens s) (port->tokens (open-input-string s))) (define (port->tokens in) (define token (lex in)) (if (eq? (position-token-token token) 'EOF) '() (cons token (port->tokens in)))) (map position-token-token (string->tokens "123*45/3")) ; strip positions ; Output: ; (list (token 'NUMBER 123) ; '* ; (token 'NUMBER 45) ; '/ ; (token 'NUMBER 3))Racket - 使用token-char定义一个字符(Racket - define one character with token-char)

我正在研究一个类的项目，我们的任务是在Python和Racket中编写数字，符号，注释，算术运算符，括号和EOF的扫描程序。 我正在研究球拍版本，我编写了以下一行来定义一个或多个角色作为符号：

[(any-char) (token-CHAR (string->character lexeme))]

我有以下行来定义一个或多个数字作为数字：

[(:+ digit) (token-NUM (string->number lexeme))]

我是Racket的新手，这是我的第三个程序，所以我不确定如何处理这个，所以任何建议都非常感谢。 我已经搜索了Racket文档，但我无法找到我想要的内容。

谢谢！

I am working on a project for a class and we are tasked with writing a scanner for numbers, symbols, comments, arithmetic operators, parenthesis, and EOF in both Python and Racket. I am working on the racket version and I have written the following line to define one or more character as a symbol:

[(any-char) (token-CHAR (string->character lexeme))]

I have the following line to define on or more digits as a number:

[(:+ digit) (token-NUM (string->number lexeme))]

I am very new to Racket, this is my third program, so I am not exactly sure how to approach this, so any suggestions are greatly appreciated. I have scoured the Racket documentation, but I wasn't able to find what I was looking for.

Thanks!

最满意答案

这是一个最小的入门示例 - 评论很多。

#lang racket ;;; IMPORT ;; Import the lexer tools (require parser-tools/yacc parser-tools/lex (prefix-in : parser-tools/lex-sre) ; names from lex-sre are prefixed with : ; to avoid name collisions syntax/readerr) ;;; REGULAR EXPRESSIONS ;; Names for regular expressions matching letters and digits. ;; Note that :or are prefixed with a : due to (prefix-in : ...) above (define-lex-abbrevs [letter (:or (:/ "a" "z") (:/ #\A #\Z) )] [digit (:/ #\0 #\9)]) ;;; TOKENS ;; Tokens such as numbers (and identifiers and strings) carry a value ;; In the example only the NUMBER token is used, but you may need more. (define-tokens value-tokens (NUMBER IDENTIFIER STRING)) ;; Tokens that don't carry a value. (define-empty-tokens op-tokens (newline := = < > + - * / ^ EOF)) ;;; LEXER ;; Here the lexer (aka the scanner) is defined. ;; The construct lexer-src-pos evaluates to a function which scans an input port ;; returning one position-token at a time. ;; A position token contains besides the actual token also source location information ;; (i.e. you can see where in the file the token was read) (define lex (lexer-src-pos [(eof) ; input: eof of file 'EOF] ; output: the symbol EOF [(:or #\tab #\space #\newline) ; input: whitespace (return-without-pos (lex input-port))] ; output: the next token ; (i.e. skip the whitespace) [#\newline ; input: newline (token-newline)] ; ouput: a newline-token ; ; note: (token-newline) returns 'newline [(:or ":=" "+" "-" "*" "/" "^" "<" ">" "=") ; input: an operator (string->symbol lexeme)] ; output: corresponding symbol [(:+ digit) ; input: digits (token-NUMBER (string->number lexeme))])) ; outout: a NUMBER token whose value is ; ; the number ; ; note: (token-value token) ; returns the number ;;; TEST (define input (open-input-string "123+456")) (lex input) ; (position-token (token 'NUMBER 123) (position 1 #f #f) (position 4 #f #f)) (lex input) ; (position-token '+ (position 4 #f #f) (position 5 #f #f)) (lex input) ; (position-token (token 'NUMBER 456) (position 5 #f #f) (position 8 #f #f)) (lex input) ; (position-token 'EOF (position 8 #f #f) (position 8 #f #f)) ;; Let's make it a little easier to play with the lexer. (define (string->tokens s) (port->tokens (open-input-string s))) (define (port->tokens in) (define token (lex in)) (if (eq? (position-token-token token) 'EOF) '() (cons token (port->tokens in)))) (map position-token-token (string->tokens "123*45/3")) ; strip positions ; Output: ; (list (token 'NUMBER 123) ; '* ; (token 'NUMBER 45) ; '/ ; (token 'NUMBER 3))

Here is a minimal getting-started example - heavily commented.

#lang racket ;;; IMPORT ;; Import the lexer tools (require parser-tools/yacc parser-tools/lex (prefix-in : parser-tools/lex-sre) ; names from lex-sre are prefixed with : ; to avoid name collisions syntax/readerr) ;;; REGULAR EXPRESSIONS ;; Names for regular expressions matching letters and digits. ;; Note that :or are prefixed with a : due to (prefix-in : ...) above (define-lex-abbrevs [letter (:or (:/ "a" "z") (:/ #\A #\Z) )] [digit (:/ #\0 #\9)]) ;;; TOKENS ;; Tokens such as numbers (and identifiers and strings) carry a value ;; In the example only the NUMBER token is used, but you may need more. (define-tokens value-tokens (NUMBER IDENTIFIER STRING)) ;; Tokens that don't carry a value. (define-empty-tokens op-tokens (newline := = < > + - * / ^ EOF)) ;;; LEXER ;; Here the lexer (aka the scanner) is defined. ;; The construct lexer-src-pos evaluates to a function which scans an input port ;; returning one position-token at a time. ;; A position token contains besides the actual token also source location information ;; (i.e. you can see where in the file the token was read) (define lex (lexer-src-pos [(eof) ; input: eof of file 'EOF] ; output: the symbol EOF [(:or #\tab #\space #\newline) ; input: whitespace (return-without-pos (lex input-port))] ; output: the next token ; (i.e. skip the whitespace) [#\newline ; input: newline (token-newline)] ; ouput: a newline-token ; ; note: (token-newline) returns 'newline [(:or ":=" "+" "-" "*" "/" "^" "<" ">" "=") ; input: an operator (string->symbol lexeme)] ; output: corresponding symbol [(:+ digit) ; input: digits (token-NUMBER (string->number lexeme))])) ; outout: a NUMBER token whose value is ; ; the number ; ; note: (token-value token) ; returns the number ;;; TEST (define input (open-input-string "123+456")) (lex input) ; (position-token (token 'NUMBER 123) (position 1 #f #f) (position 4 #f #f)) (lex input) ; (position-token '+ (position 4 #f #f) (position 5 #f #f)) (lex input) ; (position-token (token 'NUMBER 456) (position 5 #f #f) (position 8 #f #f)) (lex input) ; (position-token 'EOF (position 8 #f #f) (position 8 #f #f)) ;; Let's make it a little easier to play with the lexer. (define (string->tokens s) (port->tokens (open-input-string s))) (define (port->tokens in) (define token (lex in)) (if (eq? (position-token-token token) 'EOF) '() (cons token (port->tokens in)))) (map position-token-token (string->tokens "123*45/3")) ; strip positions ; Output: ; (list (token 'NUMBER 123) ; '* ; (token 'NUMBER 45) ; '/ ; (token 'NUMBER 3))