From 72bd20662ca836c399331375cc78d53cfc092f5f Mon Sep 17 00:00:00 2001 From: Oleg Serikov Date: Wed, 31 Oct 2018 15:01:59 +0300 Subject: [PATCH 1/4] aww the deadline :( --- 2018-komp-ling/quizzes/response-01.md | 35 +++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 2018-komp-ling/quizzes/response-01.md diff --git a/2018-komp-ling/quizzes/response-01.md b/2018-komp-ling/quizzes/response-01.md new file mode 100644 index 00000000..c4b66ed6 --- /dev/null +++ b/2018-komp-ling/quizzes/response-01.md @@ -0,0 +1,35 @@ +1. Which problems does maxmatch suffer from? (Choose all that apply.) + + *a) requires comprehensive dictionary + d) constructs non-grammatical sentences* + +2. Write a perl/sed substitution with regular expressions that + adds whitespace for segmentation around "/" in "either/or" + expressions but not around fractions "1/2": + + "either/or" -> "either / or" + "1/2" -> "1/2" + + ```sed 's/\([^0-9]\+\)\/\([^0-9]\+\)/\1 \/ \2/g'``` + +3. the text mentions several times that machine learning + techniques produce better segmentation than rule-based + systems; what are some downsides of machine learning + techniques compared to rule-based? + + *Machine-learning techniques are often harder to interpret (e.g. all these NN-based solutions). Having found the group of mistakes produced by the ML-algorithm you won't just add a bugfix cause the whole model needs to be rebuilt and retrained.* +4. write a sentence (in English or in Russian) which maxmatch + segments incorrectly. + + *An ambigous example from the school textbook: "наполеонсеялрожь". Maxmatch will choose the less statistically probable examle "наполеон сеял рожь".* + +5. what are problems for sentence segmentation? provide one + example in English or Russian for each that applies. + + a) ambiguous abbrevations with punctuation + *"... и т.п. случаи"* + b) sentences containing symbols '!' and '?' + + c) sentences lacking separating punctuation + + d) sentences not separated by whitespace From 2b740baecc1a02a2a199ffc18384e7f3ef10b5b1 Mon Sep 17 00:00:00 2001 From: Oleg Serikov Date: Wed, 31 Oct 2018 15:10:28 +0300 Subject: [PATCH 2/4] aww the deadline :( --- 2018-komp-ling/quizzes/response-01.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/2018-komp-ling/quizzes/response-01.md b/2018-komp-ling/quizzes/response-01.md index c4b66ed6..44181448 100644 --- a/2018-komp-ling/quizzes/response-01.md +++ b/2018-komp-ling/quizzes/response-01.md @@ -27,9 +27,15 @@ example in English or Russian for each that applies. a) ambiguous abbrevations with punctuation + *"... и т.п. случаи"* + b) sentences containing symbols '!' and '?' - + + "Are you O.K.?!" + + "Here ```!condition ? do_this_if_inverted_condition_is_true : do_that_if_inverted_condition_is_false``` is the ternary operator usage example" + c) sentences lacking separating punctuation d) sentences not separated by whitespace From 6499f2f01207859a8b075b09918e6ce8dcfe4ab2 Mon Sep 17 00:00:00 2001 From: Oleg Serikov Date: Wed, 31 Oct 2018 15:36:33 +0300 Subject: [PATCH 3/4] .md prettyformatting. aww the deadline :((( --- 2018-komp-ling/quizzes/response-01.md | 91 +++++++++++++++------------ 1 file changed, 50 insertions(+), 41 deletions(-) diff --git a/2018-komp-ling/quizzes/response-01.md b/2018-komp-ling/quizzes/response-01.md index 44181448..21b7e406 100644 --- a/2018-komp-ling/quizzes/response-01.md +++ b/2018-komp-ling/quizzes/response-01.md @@ -1,41 +1,50 @@ -1. Which problems does maxmatch suffer from? (Choose all that apply.) - - *a) requires comprehensive dictionary - d) constructs non-grammatical sentences* - -2. Write a perl/sed substitution with regular expressions that - adds whitespace for segmentation around "/" in "either/or" - expressions but not around fractions "1/2": - - "either/or" -> "either / or" - "1/2" -> "1/2" - - ```sed 's/\([^0-9]\+\)\/\([^0-9]\+\)/\1 \/ \2/g'``` - -3. the text mentions several times that machine learning - techniques produce better segmentation than rule-based - systems; what are some downsides of machine learning - techniques compared to rule-based? - - *Machine-learning techniques are often harder to interpret (e.g. all these NN-based solutions). Having found the group of mistakes produced by the ML-algorithm you won't just add a bugfix cause the whole model needs to be rebuilt and retrained.* -4. write a sentence (in English or in Russian) which maxmatch - segments incorrectly. - - *An ambigous example from the school textbook: "наполеонсеялрожь". Maxmatch will choose the less statistically probable examle "наполеон сеял рожь".* - -5. what are problems for sentence segmentation? provide one - example in English or Russian for each that applies. - - a) ambiguous abbrevations with punctuation - - *"... и т.п. случаи"* - - b) sentences containing symbols '!' and '?' - - "Are you O.K.?!" - - "Here ```!condition ? do_this_if_inverted_condition_is_true : do_that_if_inverted_condition_is_false``` is the ternary operator usage example" - - c) sentences lacking separating punctuation - - d) sentences not separated by whitespace +# Quiz 1 solution + + +## 1. Which problems does maxmatch suffer from? (Choose all that apply.) + +a) requires comprehensive dictionary + +d) constructs non-grammatical sentences + +--- +## 2. Write a perl/sed substitution with regular expressions that adds whitespace for segmentation around "/" in "either/or" expressions but not around fractions "1/2": + +``` +sed 's/\([^0-9]\+\)\/\([^0-9]\+\)/\1 \/ \2/g' +``` + +--- +## 3. the text mentions several times that machine learning techniques produce better segmentation than rule-based systems; what are some downsides of machine learning techniques compared to rule-based? + +Machine-learning techniques are often harder to interpret (e.g. all these NN-based solutions). Having found the group of mistakes produced by the ML-algorithm you won't just add a bugfix cause the whole model needs to be rebuilt and retrained. + +--- +## 4. write a sentence (in English or in Russian) which maxmatch segments incorrectly. + +An ambigous example from the school textbook: "наполеонсеялрожь". Maxmatch will choose the less statistically probable examle "наполеон сеял рожь". + +--- +## 5. what are problems for sentence segmentation? provide one example in English or Russian for each that applies. + +### a) ambiguous abbrevations with punctuation + +"... и т.п. случаи." + +### b) sentences containing symbols '!' and '?' + +"Are you O.K.?!" + +"Here ```!condition ? do_this_if_inverted_condition_is_true : do_that_if_inverted_condition_is_false``` is the ternary operator usage example." + +### c) sentences lacking separating punctuation + +>Hello, this is my first email +> +>And i will not use dots to finish last sentences in the paragraph. And I also love WYSIWYG +> +>Yours, me. + +### d) sentences not separated by whitespace + +"The first sentence.The second one." From 5fa054a87079bea44dc521995002550547e432ce Mon Sep 17 00:00:00 2001 From: Oleg Serikov Date: Thu, 1 Nov 2018 17:16:36 +0300 Subject: [PATCH 4/4] renamed response filename, added some notes --- .../quizzes/{response-01.md => quiz-01-response.md} | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) rename 2018-komp-ling/quizzes/{response-01.md => quiz-01-response.md} (87%) diff --git a/2018-komp-ling/quizzes/response-01.md b/2018-komp-ling/quizzes/quiz-01-response.md similarity index 87% rename from 2018-komp-ling/quizzes/response-01.md rename to 2018-komp-ling/quizzes/quiz-01-response.md index 21b7e406..55617568 100644 --- a/2018-komp-ling/quizzes/response-01.md +++ b/2018-komp-ling/quizzes/quiz-01-response.md @@ -7,6 +7,8 @@ a) requires comprehensive dictionary d) constructs non-grammatical sentences +On the performance: MaxMatch looks to be slow (*O(len(sentence) + sum(len(entry) for entry in vocabulary))*) on the worst case, but it's hard to construct such case and to meet such example in the natural language text. + --- ## 2. Write a perl/sed substitution with regular expressions that adds whitespace for segmentation around "/" in "either/or" expressions but not around fractions "1/2": @@ -26,7 +28,7 @@ An ambigous example from the school textbook: "наполеонсеялрожь" --- ## 5. what are problems for sentence segmentation? provide one example in English or Russian for each that applies. - +Depends on the segmentation algorithm. ### a) ambiguous abbrevations with punctuation "... и т.п. случаи."