This is yet another rules engine implementation attempt. Basic testing: $ prove t/re_tests.t $ prove t/rx.t ... 96.25% okay. ... 84.51% okay. $ perl -w Regexp_ModuleA.pm --test $ perl -w Regexp_ModuleA.pm --test6 ... more informative than prove, but no prove-like summary at the end. $ perl -w Regexp_ModuleA.pm --repl $ perl -w Regexp_ModuleA.pm --repl6 ... interactive. To take pugs t/regex/ tests, kludge them into p5, and run them (quickly): $ PERL5_V6_IMPLEMENTATION=PugsRegexTestA perl -w -I. -Mv6-alpha ../../../../t/regex/from_perl6_rules/anchors.t $ DEBUG_PugsRegexTestA=1 PERL5_V6_IMPLEMENTATION=PugsRegexTestA perl -w -I. -Mv6-alpha ../../../../t/regex/from_perl6_rules/anchors.t $ PERL5_V6_IMPLEMENTATION=PugsRegexTestA prove -r --perl 'perl -w -I. -Mv6-alpha' ../../../../t/regex/ To run pugs with this as its p6 regex engine: $ cp misc/pX/Common/yet_another_regex_engine/Pugs_Runtime_Match_HsBridge.pm ./perl5/Pugs-Compiler-Rule/lib/Pugs/Runtime/Match/HsBridge.pm # Running make may be required? $ ./pugs Regards forking: When you want to do an experimental fork, you usually have to do it off line, where no one else can see or play with it. That, or duplicate a big directory tree. Not here. Just copy Regexp_Module.pm to Whatever.pm. To test: RE_TESTS_MODULE=Whatever prove t/re_tests.t Ways you can help: *... Objective: Create a fast p6 rules implementation in p5 which we can use for perl6 parser development. Intent: * supports p5 and p6 regex * complete coverage, well tested * fast * quick development * written in p5 (so it can be used with pugs, v6.pm, and pil2js) * hardwired grammars - spec'ed extensibility deferred. Architecture: Uses the core Regexp::Parser::ReentrantEngine.pm emitters (fast and tested), repackaged to use a new ast (no Regexp::Parser dependency), with p5 and p6 grammars expressed as m-expressions (bootstrap both not needed, and easily done), and a minimized ast (lean on the perl5 engine, avoid unnecessary nodes and work). With compilation looking like: some_regexp -> matcher for regexp grammar -> Match tree -> m-expr tree -> ast -> matcher for some_regexp. Plan: * Update Plan, Budget. * Declare success. Mission creep: * Finish separating ast ("compile-time") from "runtime". * Source filter. PCR equivalent. (Did both:/) Bugs: (some among many) * Numerous. Running as pugs's rx engine is currently the best source. * \10 handling Budgeting: 3 days to p5 test suite passing. Roughly 1 day writing, 1 day debugging hell, 1 day to get tests green. 1 day to get p6 test framework. 1 day for simple p6 rules support. 1+fuzzy day for fleshing out rules support. So 6+ days. Likely spread over February '07. REVIEW: Currently at 15 days, going into March. Budget blown. Risks: * P6 rule spec is big. I always underestimate this. The "1+fuzzy" current estimate is silly. -> Schedule blow out. Mitigation: Break functionality into chunks (eg, related to implementations - eg, everything pge/PCR has); Re-budget; Downscale objective (eg, go for pge or pcr equivalence). REVIEW: ~11 days and unfinished. * Engine oddities, perlbug tickling. -> Exploding debug cost. Mitigation: perl version restriction. Still worrisome. REVIEW: Cost <1/2 day. * Does this version of the engine already support commits? -> significant engine change, unexpected debugging hell. Mitigation: Check; Scavenge if necessary (before initial debugging pass or after? - after - minimum path to get-it-working - worth the hit). REVIEW: Commits cost 1-2 days. * P6 rule interaction with lexical variables - have plan, but it's still raw. -> rathole. Mitigation: Flesh out plan; Punt, restricting interaction to symbol table. REVIEW: Currently sketched in but broken. * Match objects are non-spec. -> non-specness. Mitigation: I don't care; Might be easily fixed. * Implementation of grammars and rules, using packages and subs, is new development. -> schedule creep, non-specness. Mitigation: Don't really care how it's implemented, as long as it can support p6 parser development; Can even punt to flat namespace; Whatever is easy; Re non-spec, don't care unless there are tests. REVIEW: Creeped, see review. * Mission creep, eg, everything for everyone cpan rules module. Mitigation: don't; Budget separately. REVIEW: Creeped elsewise. - Performance. As of 2007-Jan-31, performance is very poor. 6 min to run re_test. Cause unexplored. Perhaps nested expressions producing really large parse trees? -> potential slow speed, unusable for parser? Mitigation: Look into performance; flatten grammars. RESOLVED: use overload; in Match was a 100x performance hit. REVIEW 2007-Mar-05 Way over time budget. Mission drift and creep. Current objective unclear. The initial objective: Pugs looked unhealthily, dangerously inactive. Audreyt was gone, pugs hs was stuck, and kp6 was moving slowly. Objective was to create a redsix-like bootstrap, standalone p6 on p5. First step was a "p5 with rules" reprise, but with p6 syntax. Environment changes, focus drifts: Audreyt's health got debugged, and is happily back. Pugs hs development resumed. So original "lifeboat" objective was obsoleted. An official p6 grammar and opp was begun, reducing the attractiveness of a redsix-like quick kludge. Engine objectives underwent "test-driven drift" toward full p6 coverage. Accomplishments to date: A 3rd p6 regex engine. For some tasks the best currently available. The p6 on top of non-reentrant p5 regexp approach could be applied to other platforms. Generated some pugs project liveliness for a couple of weeks. Motivated Perl-6.0.0-STD.pm. Current state: Code is very ugly, an artifact of deferred-refactoring minimum-delta test-driven development. In p5. Blech. Architecture seems basically sound so far. Performance is not bad, with ground-brushing low hanging regex optimization not yet applied. Where from here? (1) Punt. Original objective gone. Much better to write this in p6. Working p6 oo should be available in 1-2 months. Value as prototype isn't worth the development cost. (2) Support Perl-6.0.0-STD.pm parser development. Cease improving general p6 coverage, focus on parser. Feasibility unclear - raw p6 code is now playing a non-trivial role. Would have to transliterate. Would keep parser development from stalling over the next 1-2 months, presuming one could start running it quickly. A maybe. (3) Continue development as a p5-based p6 regex engine. Also provides reentrant p5 for pre v5.10. However PCR exists. Though slow. Mention of possible architectural difficulties. But exists. And once a p6 impl exists, would be easy to generate p5. Though that may take until Fall'07. Plausible, but doesn't greatly appeal. (4) Continue development as a pugs p6 regex engine. Which would live for a quarter or two, until a hs version replaced it. Fighting to have pugs include something which is not on its own critical/audreyt path is not fun. A maybe, but for someone else. (5) Repackage engine core as PCR backend. Thus being able to hand it off, rather than it all landing on the dusty back shelf. Could be interesting to do the backend api. But clarifying its constraints is still a work in progress. Not clear PCR ast is the place to do that. Might do minimum hack to make kp6 go faster. But how essential is kp6, with audreyt back? A maybe, but not -O fun. (6) Continue towards standalone p6 on p5. What payoffs? Insurance policy on pugs hs. Ability to modularly compose p6 implementations. The latter seems interesting, but with pugs dominant, and landing on the p5 perl6: namespace, it looks like an uphill battle. (7) Any other possible fruit? Postmortem: Was aware of scope drift, but failed to examine cause, reevaluate objectives in new context. History: Elapsed dev time: 14 days (ie, ~100 hrs). Calendar time: 37 days. 2007-Mar-05 REVIEW. 2007-Mar-05 Sigspace begun. Can now run as pugs's engine instead of PCR. re_tests.t 96%, 16 sec; rx.t 90%, 16 sec; t/regex/ Failed 54/59 test scripts. 4811/5094 subtests failed. Files=59, Tests=5094, 27 wallclock secs (21.32 cusr + 2.00 csys = 23.32 CPU) Cost: 2 days (variously squandered) 2007-Mar-01 Trivial beginning on aliases. Can now run puts t/regex/. re_tests.t 96%, 16 sec; rx.t 85%, 13 sec; t/regex/ Failed 53/58 test scripts. 4809/5089 subtests failed. Files=58, Tests=5089, 21 wallclock secs (17.68 cusr + 1.81 csys = 19.49 CPU) Cost: 1 day. 2007-02-27 Converted captures from a hack "everything is an array" to spec. Minor other rx_ test progress. re_tests.t 96%, 15 sec; rx.t 84%, 12 sec. Cost: 1 day. 2007-02-25 Engine debugging. Pondered how this module will fit into the pugs development world. Wrote some thought pieces and infrastructure for same. Cost 2 days. 2007-Feb-20 Assorted engine and p6 grammar work. Some rx test jig fixes. re_tests.t 95%, 13 sec; rx.t 75%, 11 sec. Cost: 1 day. 2007-Feb-19 REPL is now easier to use with different dialects. Commit bugfix. Began P6 captures. P5 grammar rule names changed to mirror p6 grammar. rx.pl bugfix. Began P6 support. re_tests.t 95%, 13 sec; rx.t 36%, 12 sec. Cost: 1 day. 2007-Feb-18 2007-Feb-1? More commit, but untested. Some bug fixes. Started adding a p6 test suite. 12 sec. 95% ok. 3% ok6. Cost: 1 day. Elapsed dev time: 5 days. 2007-Feb-08 2007-Feb-06 Some cleanup. Started commit. Rx's are now method-like. Added externally defined post actions. A bugfix. Started P5WithSubrules. 11 sec. 92% okay. Cost: 2 days. 2007-Feb-03 Some cleanup. Added nodes. Performance improved (7 min -> 12 sec by avoiding "use overload" when possible). 12 sec. 92% okay. Cost: 1 day. Elapsed dev time: 2+ days. 2007-Feb-01 Began backrefs. Bugfixes. Dropped in copies of the missing nodes which need to be adapted to the new ast. 400 sec. 82.00% okay. Cost: 1/4 day. 2007-Jan-31 Added non-greedy quant, and non-capturing group Performance problems (re_tests now takes 6 min). 80% okay. Cost: 1/8 day. 2007-Jan-30 --text and prove can now both be used. Added interactive repl. Added ast initialization phase. Began captures, flags. 60% okay. Cost: 1/2 day. 2007-Jan-29 Engine now running. Perlbug in v5.8.8 was disappearing a lexical (adding an extra use "fixed"). P5 grammar tweaked. Match tree to ast converter begun. First light (ie, first time through the entire "p5 re pattern -> match tree -> ast -> matcher -> matcher works" path). P5 re_tests added. 29.66% okay. Cost: 1/2 day. 2007-Jan-28 Started. Repackaged engine around new ast, emitters, and m-expr dsl. First draft of p5 rules grammar. Engine is currently broken from the repackaging. Cost: 1 day. Misc Notes: With my v5.8.8, only Devel::DProfLB didn't fail. Its granularity is very poor. #; Local Variables: #; mode: text #; End: