-
Notifications
You must be signed in to change notification settings - Fork 0
Collapse file tree
Files
Search this repository(forward slash) forward slash/
/
Copy pathLeanSERP-Studio-URL-Modedesc.txt
More file actions
More file actions
Latest commit
650 lines (517 loc) · 27.1 KB
/
Copy pathLeanSERP-Studio-URL-Modedesc.txt
File metadata and controls
650 lines (517 loc) · 27.1 KB
You must be signed in to make or propose changes
More edit options
Edit and raw actions
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
pasted-text.txt
TXT
Run this single PowerShell command:
powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -STA -File "C:\Users\PC\Downloads\LeanSERP-Studio\LeanSERP-Studio-GUI.ps1"
To open it without keeping a visible PowerShell window:
powershell
Start-Process -FilePath "powershell.exe" -ArgumentList '-NoProfile -ExecutionPolicy Bypass -STA -WindowStyle Hidden -File "C:\Users\PC\Downloads\LeanSERP-Studio\LeanSERP-Studio-GUI.ps1"' -WindowStyle Hidden
2026-07-22 15:24: powershell.exe -NoProfile -ExecutionPolicy Bypass -STA -File "C:\Users\PC\Downloads\LeanSERP-Studio\LeanSERP-Studio-GUI.ps1"
Use the reliable engine for large builds
Run important builds directly in PowerShell instead of through the GUI:
powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File "C:\Users\PC\Downloads\LeanSERP-Studio\LeanSERP-Studio.ps1" -InputFile "C:\Users\PC\Downloads\1.txt" -OutputDirectory "C:\Users\PC\Downloads\LeanSERP-Output" -ChunkSize 250000 -KeepUnderscores
Run the launcher with: powershell.exe -NoProfile -ExecutionPolicy Bypass -STA -File "C:\Users\PC\Downloads\LeanSERP-Studio\Start-LeanSERP-Build.ps1"
To test interruption safely, press Ctrl+C in that build console after Created chunk 1 appears.
safest production workflow is:
Use Start-LeanSERP-Build.ps1.
Watch progress in its dedicated console.
Press Ctrl+C to request interruption.
Verify no builder process remains.
Remove only incomplete build directories.
Use the GUI for selecting files, reports, and small builds until the polling-based runner integration is tested.
2026.07.22 18:47: "One older process, PID 7208, was still shown earlier. Since the restarted build is now running, do not terminate PowerShell processes blindly. After this build completes, verify no runner or builder remains:
powershell
$remaining = @(
Get-CimInstance Win32_Process |
Where-Object {
$_.Name -eq "powershell.exe" -and
$_.ProcessId -ne $PID -and
(
$_.CommandLine -like "*Invoke-LeanSERP-Build.ps1*" -or
$_.CommandLine -like "*LeanSERP-Studio.ps1*"
)
}
)
if ($remaining.Count -eq 0) {
Write-Host "No LeanSERP build process remains." -ForegroundColor Green
}
else {
$remaining |
Select-Object ProcessId, ParentProcessId, CommandLine |
Format-List
}
"
2026.07.23 19:22:"Press Ctrl+C only in the log-watching window after completion. Pressing it in the runner window would stop the actual build."
2026.07.27 13:11 : "pasted-text.txt
TXT
Can you understand this? What do you understand?
Yes. I understand that you are building two related but separate applications:
1. LeanSERP Studio
A standalone Windows application, currently based on PowerShell and WinForms. It should:
Use no active CPU, RAM, GPU, or network resources when closed.
Normalize, sort, deduplicate, inspect, and package very large domain or URL lists.
Keep processing work outside Firefox.
Produce compact files that LeanSERP Filter can import.
Support live progress, stopping, reports, SHA-256 verification, and bounded-memory processing.
You have two processing modes:
Label mode for inputs such as *://*.example/* or example.
URL mode for full hostnames and URLs, using the Public Suffix List to extract main labels.
2. LeanSERP Filter
A lightweight Firefox extension that will eventually:
Import the package produced by Studio.
Filter search results using exact hostname and complete-label matching.
Respect Public Suffix List boundaries.
Allow explicitly approved PSL overrides such as gov, nhs, and hluttaw.
Avoid substring matches: gov matches the complete label gov, but not mygov.
Include optional SERP simplification and resource-saving features.
Its package-import functionality has not yet been implemented.
What has been completed
Stable PowerShell label mode
The ordinary label builder works and has been tested with:
Streaming input.
Bounded sorted chunks.
Global sorting and deduplication.
BOM cleanup.
Rejection reports.
Metadata and SHA-256 verification.
Multiple files.
Live GUI logging.
Process-tree stopping and cleanup.
PowerShell URL mode
The URL normalizer works correctly but is extremely slow. On the 1,456,946-line test:
Metric Result
Accepted lines 1,455,897
Rejected lines 1,049
Unique labels 1,226,659
Exact hosts 1
PSL overrides 3
Runtime About 6,925 seconds
Its output is the correctness reference.
Compiled C# mode
The compiled normalizer processed the same 1,456,946 lines in 12.219 seconds, roughly 567 times faster.
Before applying approved rules, it produced:
1,226,658 labels.
One missing label: hluttaw.
24 extra PSL-only rejections.
The separate ApplyApprovedOverrides.exe then:
Loaded the three approved overrides.
Used two of them in that dataset.
Added the missing hluttaw label.
Used one approved exact hostname.
Produced 1,226,659 final labels.
Produced the exact same label hash as the PowerShell reference:
text
4ccb08039ea029c081f4b52200e855ddb031b7a1b491b90025d6fa3af2ca5cb0
That proves the two-stage compiled pipeline can reproduce the reference labels.txt for the benchmark.
Current compiled workflow
powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -STA -File "C:\Users\PC\Downloads\LeanSERP-Studio-URL-Mode\LeanSERP-Compiled-GUI.ps1"
To start it without a visible PowerShell console:
powershell
$arguments = @(
"-NoProfile"
"-ExecutionPolicy"
"Bypass"
"-STA"
"-WindowStyle"
"Hidden"
"-File"
"C:\Users\PC\Downloads\LeanSERP-Studio-URL-Mode\LeanSERP-Compiled-GUI.ps1"
)
Start-Process `
-FilePath "powershell.exe" `
-ArgumentList $arguments `
-WindowStyle Hidden"
2026.07.27 19:41: "The compiled normalizer removes http:// and https:// before extracting the hostname and main label.
Input Resulting label
[example.com](http://example.com/page) example
[example.com](https://www.example.com?q=1) example
[www54.example.co.uk](https://www54.example.co.uk/path) example
[subdomain.wikipedia.org](http://subdomain.wikipedia.org) wikipedia
It also removes:
Leading www, www1, www54, etc. when Remove www-number labels is checked.
Paths, queries, and fragments, such as /page, ?q=1, and #section.
Ports and credentials, where supported.
Public suffixes such as com, co.uk, and org.
The original source file is not modified. These transformations apply only while generating the normalized package. Exact approved hosts and PSL overrides remain in their separate rule files.",
2026-07-27 19:44:"The compiled normalizer creates two diagnostic reports inside each compiled-build-* directory:
File Purpose
rejected-lines.tsv Records the rejection reason, source file, line number, and original input.
public-suffix-only.tsv Records hostnames rejected because the complete hostname is a Public Suffix List entry.
For example:
text
reason source line originalpublic-suffix-only ...\input.txt 113 amplifyapp.cominvalid-url ...\input.txt 250 https://[invalidempty-comment-or-directive ...\input.txt 400 #comment
The public-suffix report contains:
text
hostname publicSuffix source linegithub.io github.io ...\input.txt 1234cloudfront.net cloudfront.net ...\input.txt 5678
Important package distinction
These reports currently remain in the intermediate compiled build, such as:
text
C:\Users\PC\Downloads\LeanSERP-Compiled-Output\compiled-build-...\rejected-lines.tsvC:\Users\PC\Downloads\LeanSERP-Compiled-Output\compiled-build-...\public-suffix-only.tsv
Your final unified package currently includes rejected-lines.tsv, but the GUI pipeline does not appear to pass the compiled rejection report to New-LeanSerpPackage.ps1. Therefore, the final package's rejected-lines.tsv is probably empty. public-suffix-only.tsv is not currently part of the unified package manifest.
You can inspect the latest compiled rejection reasons with:
powershell
$root = "C:\Users\PC\Downloads\LeanSERP-Compiled-Output"$build = Get-ChildItem -LiteralPath $root -Directory | Where-Object { $_.Name -like "compiled-build-*" } | Sort-Object LastWriteTime -Descending | Select-Object -First 1if ($null -eq $build) { throw "No compiled build was found."}$rejections = Join-Path $build.FullName "rejected-lines.tsv"if (-not (Test-Path -LiteralPath $rejections -PathType Leaf)) { throw "The rejection report was not found: $rejections"}$rows = @(Import-Csv -LiteralPath $rejections -Delimiter "`t")Write-Host "Build: $($build.FullName)"Write-Host "Rejected rows: $($rows.Count)"$rows | Group-Object -Property reason | Sort-Object -Property Count -Descending | Select-Object Count, Name | Format-Table -AutoSize
To view the first 30 rejected entries:
powershell
$rows | Select-Object -First 30 reason, source, line, original | Format-Table -Wrap -AutoSize
The next Studio improvement should copy both diagnostic files into the unified package as review-only files, without activating either one in LeanSERP Filter.",
2026.07.27 19:49: C:\Users\PC\Downloads\LeanSERP-Studio-URL-Mode\PSL-Host-Review\Reviewed\final-psl-approval-validation.txt : "hostname ruleType presentInCandidates active
github.io host-and-subdomains-block True True
"