BUG: fix writing of Categorical with to_sql (GH8624)

Use .get_values() instead of .values on the block,
this ensures that NonConsolidatable blocks (non-dense
blocks like categorical or sparse) are densified
instead of just returning the raw .values

Author: jorisvandenbossche

doc/source/categorical.rst

@@ -573,6 +573,7 @@ relevant columns back to `category` and assign the right categories and categori
     df2.dtypes
     df2["cats"]
 
+The same holds for writing to a SQL database with ``to_sql``.
 
 Missing Data
 ------------

doc/source/io.rst

@@ -3337,6 +3337,14 @@ With some databases, writing large DataFrames can result in errors due to packet
     flavors, columns with type ``timedelta64`` will be written as integer
     values as nanoseconds to the database and a warning will be raised.
 
+.. note::
+
+    Columns of ``category`` dtype will be converted to the dense representation
+    as you would get with ``np.asarray(categorical)`` (e.g. for string categories
+    this gives an array of strings).
+    Because of this, reading the database table back in does **not** generate
+    a categorical.
+
 
 Reading Tables
 ~~~~~~~~~~~~~~

doc/source/whatsnew/v0.15.1.txt

@@ -131,7 +131,7 @@ Bug Fixes
 - Bug in ``Categorical`` not created properly with ``Series.to_frame()`` (:issue:`8626`)
 - Bug in coercing in astype of a ``Categorical`` of a passed ``pd.Categorical`` (this now raises ``TypeError`` correctly), (:issue:`8626`)
 - Bug in ``cut``/``qcut`` when using ``Series`` and ``retbins=True`` (:issue:`8589`)
-
+- Bug in writing Categorical columns to an SQL database with ``to_sql`` (:issue:`8624`).
 
 
 

pandas/io/sql.py

@@ -670,7 +670,7 @@ def insert_data(self):
                 # datetime.datetime
                 d = b.values.astype('M8[us]').astype(object)
             else:
-                d = np.array(b.values, dtype=object)
+                d = np.array(b.get_values(), dtype=object)
 
             # replace NaN with None
             if b._can_hold_na:

pandas/io/tests/test_sql.py

@@ -678,6 +678,20 @@ def test_chunksize_read(self):
 
             tm.assert_frame_equal(res1, res3)
 
+    def test_categorical(self):
+        # GH8624
+        # test that categorical gets written correctly as dense column
+        df = DataFrame(
+            {'person_id': [1, 2, 3],
+             'person_name': ['John P. Doe', 'Jane Dove', 'John P. Doe']})
+        df2 = df.copy()
+        df2['person_name'] = df2['person_name'].astype('category')
+
+        df2.to_sql('test_categorical', self.conn, index=False)
+        res = sql.read_sql_query('SELECT * FROM test_categorical', self.conn)
+
+        tm.assert_frame_equal(res, df)
+
 
 class TestSQLApi(_TestSQLApi):
     """