main commit, files and docs

METHOD.md

@@ -0,0 +1,56 @@
+# Method
+
+These files were obtained using reconnecting bots.
+
+I have observed that while connecting through an IP that is not reputable (e.g, a VPN or proxy IP), users are shown a captcha when they log in. This captcha looks a bit like this:
+
+![A test image of a captcha](images/captcha.png)
+
+These captchas are a bit funny, because they can be solved easier using a bot than by a human. However, I noticed that other AI-based solutions, such as Tesseract (OCR) or even paid services such as NopeCHA consistently provide an incorrect or invalid result. Therefore, I set out to create a captcha solving AI model for this!
+
+# Data collection
+
+Data was collected using Mineflayer bots on NordVPN to consistently get Minehut to offer the client captchas. Mineflayer doesn't provide a native interface to manage maps, so I had to get a bit creative. Minecraft uses a limited set of colors, so I found a premade map between colors that the server offers and hexadecimal colors that can be parsed in an image. 248 colors were mapped.
+
+Maps are in item frames, so they had to be combined twice, first vertically, then horizontally. They were assigned UUIDS.
+
+# Classification
+
+Technically I could have manually classified these files, but I instead paid a few dollars to get human classifiers at [2Captcha](https://2captcha.com). 500 images were downloaded by the bot, 438 of them were usable. Human classifiers identified and classified 410 files (some captchas are unsolvable because the letters are too distorted).
+
+# Neural Network
+
+Tensorflow (Keras) was used to train the neural network. A [Convolutional Neural Network](https://en.wikipedia.org/wiki/Convolutional_neural_network) was used to train this neural network. RELU was chosen as an activation function, because sigmoid correctly identified the characters about half of the time.
+
+Images and strings (labels) were encoded using Numpy.
+```python
+def convert_text_to_array(text, char_to_idx, max_len):
+    text_array = [char_to_idx[char] for char in text]
+    padded_text_array = np.pad(text_array, (0, max_len - len(text_array)), 'constant')
+    return padded_text_array
+
+def convert_image_to_array(image_path, width, height):
+    img = Image.open(image_path).resize((width, height))
+    img_array = np.asarray(img) / 255.0
+    return img_array
+```
+
+# Prediction
+
+Prediction is simple: the model predicts what the characters are given an image that is passed in, and that is converted back to a string.
+
+# Downloading
+
+Models are available for download in [the files folder](./files/models)
+
+# Running
+
+A Java and Javascript (Mineflayer) client is coming soon, as well as a C# API (for use in OQMinebot and MinecraftConsoleClient).
+
+For now, it runs in python/google colab well. I haven't yet tested it on a cpu-only machine, but it's light enough to work well on one, with a few fixes.
+
+# Why?
+
+This project was made for fun, and to showcase how easily captchas can be broken. Timing myself, I solve captchas in about 8 seconds. The AI can solve it in 22-25ms. Please don't use image captchas to secure anything, they've been outdated for years now.
+
+I provide no guarantee for support, but you can hire me for projects at Incomprehensibl#9989, @JIBSIL (Telegram) or tech101file@gmail.com, whichever you prefer. If something breaks I will do my best to fix it, just open an issue on Github or here.

data_collection/README.md

@@ -0,0 +1,12 @@
+## Data collection
+
+Read more about this in [the method README](../METHOD.md)
+
+This contains the files needed to collect the data.
+
+## Manifest:
+- getData.js: Follow the instructions at the top of the file. Outputs unfiltered data in the `data` folder
+- colors.json: Color map between numbers (what the Minecraft server replies with for map data) and hex (needed to write to a png image)
+- recognize.js: Classifies and labels the images. Outputs to data.json. You can find a pretrained data.json to go with the pregenerated dataset [here](../files/data.json)
+- sort.js: Small cleaning script, to remove entries that aren't 5 letters and aren't ERROR (ERROR is output if the captcha couldn't be solved in time). It outputs data-filtered.json
+- rename.js: Renames files so they're compatible with the training process easily (removes the requirement for an external classification JSON file, as it renames the files to what the label is)

data_collection/colors.json

@@ -0,0 +1,250 @@
+{
+    "0": "#00000000",
+    "1": "#00000000",
+    "2": "#00000000",
+    "3": "#00000000",
+    "4": "#FF597d27",
+    "5": "#FF6d9930",
+    "6": "#FF7fb238",
+    "7": "#FF435e1d",
+    "8": "#FFaea473",
+    "9": "#FFd5c98c",
+    "10": "#FFf7e9a3",
+    "11": "#FF827b56",
+    "12": "#FF8c8c8c",
+    "13": "#FFababab",
+    "14": "#FFc7c7c7",
+    "15": "#FF696969",
+    "16": "#FFb40000",
+    "17": "#FFdc0000",
+    "18": "#FFff0000",
+    "19": "#FF870000",
+    "20": "#FF7070b4",
+    "21": "#FF8a8adc",
+    "22": "#FFa0a0ff",
+    "23": "#FF545487",
+    "24": "#FF757575",
+    "25": "#FF909090",
+    "26": "#FFa7a7a7",
+    "27": "#FF585858",
+    "28": "#FF005700",
+    "29": "#FF006a00",
+    "30": "#FF007c00",
+    "31": "#FF004100",
+    "32": "#FFb4b4b4",
+    "33": "#FFdcdcdc",
+    "34": "#FFffffff",
+    "35": "#FF878787",
+    "36": "#FF737681",
+    "37": "#FF8d909e",
+    "38": "#FFa4a8b8",
+    "39": "#FF565861",
+    "40": "#FF6a4c36",
+    "41": "#FF825e42",
+    "42": "#FF976d4d",
+    "43": "#FF4f3928",
+    "44": "#FF4f4f4f",
+    "45": "#FF606060",
+    "46": "#FF707070",
+    "47": "#FF3b3b3b",
+    "48": "#FF2d2db4",
+    "49": "#FF3737dc",
+    "50": "#FF4040ff",
+    "51": "#FF212187",
+    "52": "#FF645432",
+    "53": "#FF7b663e",
+    "54": "#FF8f7748",
+    "55": "#FF4b3f26",
+    "56": "#FFb4b1ac",
+    "57": "#FFdcd9d3",
+    "58": "#FFfffcf5",
+    "59": "#FF878581",
+    "60": "#FF985924",
+    "61": "#FFba6d2c",
+    "62": "#FFd87f33",
+    "63": "#FF72431b",
+    "64": "#FF7d3598",
+    "65": "#FF9941ba",
+    "66": "#FFb24cd8",
+    "67": "#FF5e2872",
+    "68": "#FF486c98",
+    "69": "#FF5884ba",
+    "70": "#FF6699d8",
+    "71": "#FF365172",
+    "72": "#FFa1a124",
+    "73": "#FFc5c52c",
+    "74": "#FFe5e533",
+    "75": "#FF79791b",
+    "76": "#FF599011",
+    "77": "#FF6db015",
+    "78": "#FF7fcc19",
+    "79": "#FF436c0d",
+    "80": "#FFaa5974",
+    "81": "#FFd06d8e",
+    "82": "#FFf27fa5",
+    "83": "#FF804357",
+    "84": "#FF353535",
+    "85": "#FF414141",
+    "86": "#FF4c4c4c",
+    "87": "#FF282828",
+    "88": "#FF6c6c6c",
+    "89": "#FF848484",
+    "90": "#FF999999",
+    "91": "#FF515151",
+    "92": "#FF35596c",
+    "93": "#FF416d84",
+    "94": "#FF4c7f99",
+    "95": "#FF284351",
+    "96": "#FF592c7d",
+    "97": "#FF6d3699",
+    "98": "#FF7f3fb2",
+    "99": "#FF43215e",
+    "100": "#FF24357d",
+    "101": "#FF2c4199",
+    "102": "#FF334cb2",
+    "103": "#FF1b285e",
+    "104": "#FF483524",
+    "105": "#FF58412c",
+    "106": "#FF664c33",
+    "107": "#FF36281b",
+    "108": "#FF485924",
+    "109": "#FF586d2c",
+    "110": "#FF667f33",
+    "111": "#FF36431b",
+    "112": "#FF6c2424",
+    "113": "#FF842c2c",
+    "114": "#FF993333",
+    "115": "#FF511b1b",
+    "116": "#FF111111",
+    "117": "#FF151515",
+    "118": "#FF191919",
+    "119": "#FF0d0d0d",
+    "120": "#FFb0a836",
+    "121": "#FFd7cd42",
+    "122": "#FFfaee4d",
+    "123": "#FF847e28",
+    "124": "#FF409a96",
+    "125": "#FF4fbcb7",
+    "126": "#FF5cdbd5",
+    "127": "#FF307370",
+    "128": "#FF345ab4",
+    "129": "#FF3f6edc",
+    "130": "#FF4a80ff",
+    "131": "#FF274387",
+    "132": "#FF009928",
+    "133": "#FF00bb32",
+    "134": "#FF00d93a",
+    "135": "#FF00721e",
+    "136": "#FF5b3c22",
+    "137": "#FF6f4a2a",
+    "138": "#FF815631",
+    "139": "#FF442d19",
+    "140": "#FF4f0100",
+    "141": "#FF600100",
+    "142": "#FF700200",
+    "143": "#FF3b0100",
+    "144": "#FF937c71",
+    "145": "#FFb4988a",
+    "146": "#FFd1b1a1",
+    "147": "#FF6e5d55",
+    "148": "#FF703919",
+    "149": "#FF89461f",
+    "150": "#FF9f5224",
+    "151": "#FF542b13",
+    "152": "#FF693d4c",
+    "153": "#FF804b5d",
+    "154": "#FF95576c",
+    "155": "#FF4e2e39",
+    "156": "#FF4f4c61",
+    "157": "#FF605d77",
+    "158": "#FF706c8a",
+    "159": "#FF3b3949",
+    "160": "#FF835d19",
+    "161": "#FFa0721f",
+    "162": "#FFba8524",
+    "163": "#FF624613",
+    "164": "#FF485225",
+    "165": "#FF58642d",
+    "166": "#FF677535",
+    "167": "#FF363d1c",
+    "168": "#FF703637",
+    "169": "#FF8a4243",
+    "170": "#FFa04d4e",
+    "171": "#FF542829",
+    "172": "#FF281c18",
+    "173": "#FF31231e",
+    "174": "#FF392923",
+    "175": "#FF1e1512",
+    "176": "#FF5f4b45",
+    "177": "#FF745c54",
+    "178": "#FF876b62",
+    "179": "#FF473833",
+    "180": "#FF3d4040",
+    "181": "#FF4b4f4f",
+    "182": "#FF575c5c",
+    "183": "#FF2e3030",
+    "184": "#FF56333e",
+    "185": "#FF693e4b",
+    "186": "#FF7a4958",
+    "187": "#FF40262e",
+    "188": "#FF352b40",
+    "189": "#FF41354f",
+    "190": "#FF4c3e5c",
+    "191": "#FF282030",
+    "192": "#FF352318",
+    "193": "#FF412b1e",
+    "194": "#FF4c3223",
+    "195": "#FF281a12",
+    "196": "#FF35391d",
+    "197": "#FF414624",
+    "198": "#FF4c522a",
+    "199": "#FF282b16",
+    "200": "#FF642a20",
+    "201": "#FF7a3327",
+    "202": "#FF8e3c2e",
+    "203": "#FF4b1f18",
+    "204": "#FF1a0f0b",
+    "205": "#FF1f120d",
+    "206": "#FF251610",
+    "207": "#FF130b08",
+    "208": "#FF852122",
+    "209": "#FFa3292a",
+    "210": "#FFbd3031",
+    "211": "#FF641919",
+    "212": "#FF682c44",
+    "213": "#FF7f3653",
+    "214": "#FF943f61",
+    "215": "#FF4e2133",
+    "216": "#FF401114",
+    "217": "#FF4f1519",
+    "218": "#FF5c191d",
+    "219": "#FF300d0f",
+    "220": "#FF0f585e",
+    "221": "#FF126c73",
+    "222": "#FF167e86",
+    "223": "#FF0b4246",
+    "224": "#FF286462",
+    "225": "#FF327a78",
+    "226": "#FF3a8e8c",
+    "227": "#FF1e4b4a",
+    "228": "#FF3c1f2b",
+    "229": "#FF4a2535",
+    "230": "#FF562c3e",
+    "231": "#FF2d1720",
+    "232": "#FF0e7f5d",
+    "233": "#FF119b72",
+    "234": "#FF14b485",
+    "235": "#FF0a5f46",
+    "236": "#FF464646",
+    "237": "#FF565656",
+    "238": "#FF646464",
+    "239": "#FF343434",
+    "240": "#FF987b67",
+    "241": "#FFba967e",
+    "242": "#FFd8af93",
+    "243": "#FF725c4d",
+    "244": "#FF597569",
+    "245": "#FF6d9081",
+    "246": "#FF7fa796",
+    "247": "#FF43584f"
+  }

data_collection/getData.js

@@ -0,0 +1,134 @@
+/*
+
+IMPORTANT NOTE:
+To run, do `yarn install` or `npm install` in your console
+Then, replace your_email_here with your microsoft account on line 44
+
+*/
+
+const mineflayer = require('mineflayer')
+const sharp = require('sharp')
+const mergeImg = require('merge-img')
+const fs = require('fs');
+const path = require('path');
+const { v4: uuidv4 } = require('uuid');
+
+function combineVert(uuid) {
+    return new Promise((resolve, reject) => {
+        loop = 0
+        for (let i = 0; i < 10; i++) {
+            if (i % 2 === 0) { // if number is even
+                let iterations = loop
+                loop++
+                mergeImg([path.join(__dirname, uuid, `map${i}.png`), path.join(__dirname, uuid, `map${i + 1}.png`)], {
+                    direction: true
+                }).then((img) => {
+                    img.write(path.join(__dirname, uuid, `${iterations}.png`), () => {
+                        if (i === 8) {
+                            resolve()
+                        }
+                    });
+                });
+            }
+        }
+    })
+}
+
+function run() {
+    return new Promise((resolve) => {
+        const uuid = `data${uuidv4()}`;
+        let num = 0;
+        fs.mkdirSync(`./${uuid}`)
+
+        const bot = mineflayer.createBot({
+            username: "your_email_here",
+            host: 'minehut.com',
+            viewDistance: 'tiny',
+            auth: 'microsoft',
+            version: '1.17.1'
+        })
+
+        bot._client.on('map', async (map) => {
+            let data = map.data
+
+            function hexToByte(hex) {
+                debugger
+                if (hex.charAt(0) == '#') {
+                    hex = hex.substr(1)
+                }
+                const a = parseInt(hex.slice(0, 2), 16)
+                const r = parseInt(hex.slice(2, 4), 16)
+                const g = parseInt(hex.slice(4, 6), 16)
+                const b = parseInt(hex.slice(6, 8), 16)
+                return [r, g, b, a]
+            }
+
+            const buf = new Buffer.from(data)
+            const imgBuf = new Uint8ClampedArray(128 * 128 * 4)
+            for (let index = 0; index < imgBuf.byteLength; index += 4) {
+                const colorMap = require('./colors.json')
+                const colorArr = hexToByte(colorMap[buf[index / 4]])
+                for (let k = 0; k < 4; k++) {
+                    imgBuf[index + k] = colorArr[k]
+                }
+            }
+            let mapbuffer = await sharp(imgBuf, {
+                raw: {
+                    width: 128,
+                    height: 128,
+                    channels: 4
+                }
+            })
+                .png()
+                .toBuffer()
+            fs.writeFileSync(path.join(__dirname, uuid, `map${num}.png`), mapbuffer)
+            if (num === 9) {
+                async function combineMaps() {
+                    await combineVert(uuid)
+                    mergeImg([path.join(__dirname, uuid, `0.png`), path.join(__dirname, uuid, `1.png`), path.join(__dirname, uuid, `2.png`), path.join(__dirname, uuid, `3.png`), path.join(__dirname, uuid, `4.png`)], {
+                        direction: false
+                    }).then((img) => {
+                        img.write(path.join(__dirname, uuid, "resultraw.png"), () => {
+                            for (let i = 0; i < 5; i++) {
+                                fs.unlinkSync(path.join(__dirname, uuid, `${i}.png`))
+                            }
+
+                            for (let i = 0; i < 10; i++) {
+                                fs.unlinkSync(path.join(__dirname, uuid, `map${i}.png`))
+                            }
+                            bot.end();
+                            resolve();
+                        });
+                    }).catch((err) => {
+                        console.log(`Error: ${err}`)
+                    })
+                };
+                combineMaps()
+            }
+            num++
+        })
+
+        bot.on('error', (err) => console.log(err))
+        bot.on('kicked', reason => console.info('Kicked for', reason))
+        bot.on('end', (reason) => console.log('Disconnected ' + reason))
+    })
+}
+
+function sleep(m) {
+    return new Promise((resolve) => {
+        setTimeout(() => {
+            resolve()
+        }, m)
+    })
+}
+
+async function main() {
+    for (let i = 0; i < 100; i++) {
+        await run();
+        await sleep(5000)
+        console.log('completed ' + i)
+    }
+    process.exit(0)
+};
+
+main()

data_collection/recognize.js

@@ -0,0 +1,60 @@
+// IMPORTANT NOTE: get a captcha key and refill your account on 2captcha.com, then place the key on line 6
+
+const fs = require('fs');
+const path = require('path');
+const Captcha = require("2captcha")
+const solver = new Captcha.Solver("YOUR_2CAPTCHA_KEY_HERE")
+const sharp = require('sharp');
+
+const directories = [];
+const completedImages = [];
+
+const allFiles = fs.readdirSync('raw');
+
+function process(path) {
+    return new Promise(async (resolve) => {
+        const imageBuf = await sharp(`raw/${path}/resultraw.png`)
+            .resize({
+                width: 550,
+                height: 256
+            })
+            .toBuffer();
+        const base64string = `data:image/png;base64,${imageBuf.toString('base64')}`
+        solver.imageCaptcha(base64string)
+            .then((res) => {
+                resolve(res.data);
+            })
+    })
+}
+
+async function main() {
+    for (let i = 0; i < allFiles.length; i++) {
+        if (!allFiles[i].split('data')[0]) {
+            const file = allFiles[i];
+            const filepath = path.join(__dirname, 'raw', file, 'resultraw.png')
+            const fileExists = fs.existsSync(filepath);
+            if (fileExists) {
+                directories.push(file);
+            } else {
+                console.log('Skipped directory ' + file + ' for reason: invalid format');
+                console.log('Cleaning dataset of this invalid entry!');
+                fs.rmSync(path.join(__dirname, 'raw', file), { recursive: true, force: true })
+            }
+        }
+    };
+
+    for(let i = 0; i < directories.length; i++) {
+        const solved = await process(directories[i]);
+        console.log('Solved captcha ' + i + ' with result ' + solved)
+        completedImages.push({
+            uuid: directories[i],
+            solution: solved
+        });
+        if(fs.existsSync('data.json')) {
+            fs.unlinkSync('data.json')
+        }
+        fs.writeFileSync('data.json', JSON.stringify(completedImages))
+    }
+};
+
+main()

data_collection/rename.js

@@ -0,0 +1,19 @@
+// rename to a format compatible with prepareData
+
+const fs = require('fs');
+
+const data = JSON.parse(fs.readFileSync('data.json'));
+const dir = 'raw'
+
+if(!fs.existsSync('data')) fs.mkdirSync('data');
+
+for(let i = 0; i < data.length; i++) {
+    const datai = data[i]
+    try {
+        fs.copyFileSync(`${dir}/${datai.uuid}/resultraw.png`, `data/${datai.solution}.png`)
+    } catch(e) {
+        console.log(e)
+    }
+}
+
+console.log('done')

data_collection/sort.js

@@ -0,0 +1,12 @@
+const fs = require('fs');
+
+let file = JSON.parse(fs.readFileSync('data.json'));
+const oldLen = file.length;
+
+file = file.filter((value, index, array) => {
+  return (value.solution.length === 5 && value.solution !== 'ERROR')
+})
+console.log(`Filtered to ${file.length} items from ${oldLen} items`)
+
+fs.writeFileSync('data-filtered.json', JSON.stringify(file))
+console.log('done')

files/README.md

@@ -0,0 +1,14 @@
+## This directory is used in getting needed files via Colab
+
+For those who don't know, Colab refers to [Google Colab](colab.research.google.com), which is a service that gives you free GPU and TPU (GPUs but optimized for AI research) credits to anyone with a Google account.
+
+## Manifest:
+- raw: directory of raw files gotten through MC clients. These are not labelled or cleaned!
+- data: directory of data files used while training the model
+- models: pretrained models
+- data.json: JSON file of labels, mapping raw/* files to labels.
+- raw.zip: raw directory, but zipped for easier downloading
+- data.zip: data directory, but zipped so downloads in colab go faster
+
+## Files:
+Want to know how these files were obtained? This is documented with a lot of detail in [METHOD.md](../METHOD.md)